I assume that there is no outside speaker already, as we have on ours, which is surprising considering you have the 3-way switch already. But given that, there should be a mfgr. wiring diagram that came with your trailer's stereo, or that you can get from the mfgr, which would be invaluable for this project.
But, my main point is that there may be a better pathway to the outside via the back side of the refrigerator, if that is close by your stereo, as our is. And there are already multiple low voltage wires that come from the refrigerator controller board located on the outside into the refrigerator, so you would not have to drill a hole.
The drawback of this approach is that, while you don't have to drill a hole in the exterior of your trailer, there will be a visible cable or wire leading from your refrigerator's vent cover on the outside. However, if you use twisted pair jacketed cable it may not be that noticable.
Just a thought.