It is true that sex is sometimes more complicated than male and female.
There are quite a lot of people with intersex disorders of the sex organs and/or sex chromosomes.
People we are calling transgender now typically aren't intersex however.
When people call out that sex is solely determined by the presence or lack of a Y chromosome, I tend to bring up complete androgen insensitivity. People with it are genetically male, but due to the receptor being broken, the androgens flip around to estrogens or such, and what you get is somebody who is
extremely female on the outside, including outlook.
The only way to detect the condition short of a DNA test(the easiest way these days) is a rather close examination by somebody who really knows what to look for. Initial signs are no periods, but as you might realize, that could be a couple hundred things.
These days, often detected early by standard paternity tests coming back "Yes, he's the father of the boy" and visual inspection showing female bits.
Note, androgen insensitivity isn't always complete. So in some cases you might have to observe some to see if the baby is more "boy" or "girl". But here we're looking at an actual medical condition readily identified through examination and a bit of hormonal/DNA testing.
Most transgenders, as you say, are NOT intersex in any fashion. They're 100% functional examples of their original gender. People with CAIS are genetically male, 99% physically and mentally female, just sterile.
So yeah, while I'll recognize various intersex conditions, I'm really on edge about transexuals. If only there was a magic pill to make them feel secure in their own skin.