Not sure why you cannot flip the 120 switch to Off, but I do know they tell you to not run the fridge on two different modes at the same time. Possibly with both 12V and 120V on at the same time something got screwed up in the controller.
As for how long it takes to cool, from my experience, 12V is the slowest to cool; and 120V and Propane are the fastest. I cool the fridge down on 120V the day before a trip, the run 12V while traveling, then switch to propane as soon as I arrive at the site (I very seldom have any power where I camp).
Might try to turn the 120V control knob, I believe counterclockwise, until it clicks to zero cooling and see if that helps.
Trailer: Lifted 228BH, heavy duty springs and Yokohama tires
TV: 2016 GMC Sierra Z71 4x4 CC, SLT
Spare TV: Two Alaskan Malamutes
Living somewhere in ID; previously lived in Moab UT; previous to that, don't ask!