The thing is, jackets and ties and dresses used to be "everyday clothes", formal was white tie and tux/evening gown. The stuff we wear as casual clothes today used to be work uniforms for cowboys and stevedores and even they changed out of it once they got home.
I wear a suit and usually a tie to work (mortgage banker so I fit in the two categories) and I just bought a black fedora to wear with my suits and overcoat. Now I wear the hat at all times when outdoors. One, it keeps my head warm, two, I get nothing but complements and "that looks good, I wish hats were back in style" comments from all ages and, three, I feel funny uncovered outside (thanks USMC) but don't like wearing ballcaps (a sports uniform item) when not playing a sport. I'll probably get a brown one for my more casual days and to go with my leather jackets.
A nice hat really finishes off an outfit. I will wear jeans (clean and not worn out) day to day but since I was in the Marines I still can't go out without a belt, real shoes and a collared shirt unless I'm on a beach or PT-ing.
I look at old pictures of factory workers with shirts and ties on under their coveralls and people in suits and hats in soup lines in the 30's and wonder how dressing well got to be such a hassle.
One cause I have thought of was the 60's-70's "tight fit" and polyester thing. Men's pants used to have pleats and lots of room to move, jackets and shirts were cut fuller and all were from natural fabrics, not plastics that don't breathe. If your formative experience with a suit was being jammed into a poorly made, poorly fitted, off the rack Sear's polyester special just to go to boring church once a week you aren't going to appreciate properly made more formal clothing when you grow up.