I also have no problem going back and forth. I wonder how much of the problem is the part of the brain that does math vs the part of the brain that does visual perception? I have one of those nerd clocks I used to keep in my office at work that shows each analog number as a math formula. I had a few instances of people who could read an analog clock that would hesitate when they looked at it to check the time.
I always thought that was interesting, because to me, telling time on an analog clock is all about knowing the position of the hands and has nothing to do with the numbers. In German, it's common to say, "quarter past three", but instead of saying "quarter to four", you say, "three quarters of four". Though you can also say something like "fifteen hours forty five".