In a perfect world, you would ideally be able to lock all 4 wheels on gravel (IMO). But more commonly (for me), I can lock one axle or the other at normal settings. I always hypothesize that it is because the axle loads aren't identical; even with a pretty level trailer.
I had exactly your situation this summer. So, I bit the bullet and went under the trailer. Jacked up one side at a time (with jack stands!) pulled the plug on the back side of the drum, used a brake adjusting tool ($3) to adjust the brakes to make better contact. Made a big difference.
There are threads on the forum about adjusting brakes if you do a search. Good luck!
2012 F150 FX4 Ecoboost, 2016 Surveyor 274BHS
2003 Yamaha FJR1300, Demco Premiere Slider
1969 John Deere 1020, 1940 Ford 9N, 1948 Ford 8N
Jonsered 535, Can of WD-40, Duct Tape
Red Green coffee mugs