It'll likely depend on what you have in the Highlander. What towing package, etc. Figure on the R-Pod 178 weighing about 3500 lbs (empty plus some stuff), with about 320-350 lbs for the hitch weight.
If it is close, you can do it. Just be aware that you will putting extra wear and tear on the vehicle (transmission problems mostly). However, it is up to you. I am not a deputized member of the Weight Police, so won't tell you you're crazy!
Here is a pretty good article for explaining your towing capacity.
Trailering 101 - What Can My Vehicle Tow?
If you're good at math, do a search for the towing capacity formulas, find the yellow sticker somewhere in the Highlander's drivers'-side door frame, and figure it out. Your towing capacity will likely be 2000-3000 lbs, depending on what that sticker says. This means you are at or a little bit over your capacity (assuming you have 3000 lbs). However, that doesn't mean you have to despair. Plenty of people do it. Do they enjoy towing in the configuration? Not usually. However, it all depends on how far you're going, how fast you plan to go, are their mountains involved, Weight distribution hitch configuration, etc. The final decision is up to you.
Good luck with you're new home-away-from-home! I hope y'all have lots of fun times with it.