I've never needed to fiddle with our current trailer's coupler, but on our previous (Coleman popup) I did. The latching mechanism had an adjustment nut for fine-tuning the fit around the underside of the ball.
If you have this adjustment (not sure if all couplers do), you may find that yours is so tight, that it's actually hard-tight on the ball. If so, not only would this be difficult to unlatch, you may be wearing the coupler and latch prematurely.
See if you have an adjuster, and try loosening just a bit, you want it as tight as possible, without creating the issues you are describing.
PS - My problem was the opposite on the popup. It was too loose. I'd noticed that with some upward pressure from the dolly, the coupler would release with the latch only slightly lifted. I tightened it up until it was difficult to get on the ball, then backed off a tad.
thebrakeman ('70), DW ('71), DD ('99), DD ('01), DD ('05)
2004 Surveyor SV261T (UltraLite Bunkhouse Hybrid)
2006 Mercury Mountaineer V8 AWD Premier
Equal-i-zer WDH (10k), Prodigy Brake Controller