Not to be negative nelly but, I wouldn't try it. My ladder has 8 anchor places with 2 screws each place. I believe the twisting moment on the expansion anchors would be enough to rip through the very thin fiberglas siding.
If I were bound and determined to add one I'd get some finished aluminum profile and through bolt the ladder to it, placing the profile on the inside of the RV wall. It wouldn't look too bad and would support the weight required.
There are many different profiles but something like this wouldn't look bad.
Aluminum Profile