As several have pointed out, the most important consideration is being certain the size/weight of the TT is well within the limits of your TV. In my experiences, the TT dealer will discount the legitimate abilities of your TV, offering that it can pull just about anything on their lot. And, on their lot, that may be true! Obviously, that's not the real world. You are wise to consider your overall abilities, whether the TV, your personal trailering abilities, or size of unit.
Given a family of four and large 5th companion, unless you plan on camping a couple weekends a year, a larger unit is what most would prefer. I have a 29' (that's actually 30' -- 36' hitch to spare) and for wife, myself, and our black lab, it's a good size. Any smaller and we would not have been happy. But that's us! What's important is what works for your family.
Remember, in the end, what's important is your safety and the safety of all of those around you -- family and otherwise. Better to purchase a unit your are comfortable with now, and trade up in time, than to buy one so large it takes the fun out of the experience.
Wishing you great success!