"If I leave it higher all the time, things can get damp."
Yep, exactly the same for my place.
I have a huge dehumidifier running pretty much all the time in the basement (primarily to protect my gun safes) and a smaller one upstairs that I use during and after showers.
Even with those it's not uncommon for the humidity in my place to go well in excess of 65%, and no matter what the temperature, that's just uncomfortable as all hell.