For future reference, this works well. I used a board on the bottom so that the TV can use its normal stand. The board has a profile that matches the little lip at the bottom of the shelf. I have some small blocks of wood in the back with velcro that hold the TV to the wall. It is stable just sitting there on its own, but it would not take much to make it tip out. The velcro is just to make sure it stays put until I want to remove it for transport, or to move it over to the bedroom side.
I'm thinking of getting rid of the velcro and just mounting the board somehow to the bottom shelf. Then the TV will be easy to just lift out of the hole when I want to move it.
Only thing left is to finish the board at the bottom and figure out how to secure it to the shelf.