Island House - x2! BJ, the owner, runs a tight ship. This would be THE place I'd steer you to and where we go each time we stay at that TT CG.
I will also second Bill's Prime for surf and turf/raw bar, but will suggest you steer clear of Don's! I've spent time all over, so slow service doesn't chaff me like others, but your service there is always slow. However, I will not tolerate improperly prepared seafood especially if we're talking spoiled oysters!
Sugarbakers is just evil and I cannot resist the sticky buns, and they excuse the often lousy attitude behind the counter. Skip their coffee unless you'll drink anything called coffee.
When at Virginia Landing, we tend to just be R&Ring and it's a place for it! If you like fishing, enough said. There's the little railroad museum to check out, and kayaks can be rented at Southeast Expeditions. You might want to check out Tangier Island too. There are tours and you can rent bikes (golf carts I think too) to explore the completely crabbing-oriented island. Check out Four Brothers Crab House and Ice Cream - all I need to say!
Enjoy your stay!