my first thought, as ive seen this kind of thing before is a dirty carb on the genset. under light load the 'low speed' jet being partially plugged causes a lean out which causes the engine speed to surge (albeit it can be hardly audible, a laser/photo tach is nice). when the generator is under load, the throttle opens and the effect of the partially plugged low speed jet is negated as it has little influence at that point.
Short of this, if the generator is running 100% with no engine surging, the AVR could be on its way out the door.
'14 XLR 395AMP
'16 Ram 3500 DRW, 6.7 cummins w/aisin trans and 4.10 gears