I'd agree with the "once a year" rotation theory. Honestly, for a well kept gun and holster that's not worn in extremely wet or corrosive environments, once a year is probably too conservative in reality. However, better safe than sorry IMO.
Odds are with quality factory ammo, if the ammo is bright and clean, and not visibly deformed, gouged, or the round is seated deeper than it's brothers, it's going to fire. However that's the odds. Since it's something under your control, might as well make those odds stack in your favor as much as possible. Honestly, in Mike's scenario of being loaded in a gun with oil, if the bullet is finger tight or better in the case, and the primer is properly seated, I'd say the gun would have to be dripping wet with low viscosity high-VOC petroleum distillates.
Even then, actively trying to disable reloading components like spilled powder, and the occasional bent/crushed primer with stuff like WD-40, direct application, and a soak for 72 hours on my workbench in a jar, some powders still burned vigorously, and the primer I placed under a sledge hammer head in my garage and whacked it, still went off. I just throw powder on my lawn, and unwanted/wrecked primers in trash that's not going to get crushed or burned.
The biggest real world potential problem with carry ammo is if the pistol is a semi-auto and unloaded/reloaded with any frequency, the top round in the magazine or one that's going into the chamber can get some setback in the case, creating potentially dangerous pressure, or perhaps inconsistent pressure that could mess up cycling and ejection causing a malfunction, stovepipe jam etc. when you least want it.
So I say once a year is good. Just in case.
If it's slipped your mind and it's been a handful of years, if the ammo is good and clean looking, you were more than likely fine all that time too. However might as well replace it on better safe than sorry aspect of the thing.
Bigger than the ammo life is probably just keeping things cycling so the weird oddball hang-up or rough patch, or lube-free dry patch of freak metal friction isn't hanging up somewhere in the gun, or ammo stack in the magazine.