Personally, I would not trust the ladder to hold a kayak. If you look at the size of the screws holding the ladder to the back of the rig, you'll see they are fairly small and not necessarily going into anything other than the siding. When you figure the additional stress and load created by the normal jostling of going down the road, that adds quite a bit of strain on the ladder. I would consider adding additional support from the rear bumper to carry more of the load.
Can you imagine what would happen if something came loose while traveling at 50+ mph? The poor guy behind you