The other unspoken part in it is that if you dig too deep into why Jesus only chose men as his apostles. It runs dangerously close to heresy which threatens to debase Christianity as a whole. Not of course that it's actually "threatened" per-se, but it's definitely unseemly, and a topic they'd rather not give any credence to, even if it's only acknowledging it to for the purpose of a rebuke.
The church can and would obviously counter with "God's Will", "God's Plan", etc. but it creates an opening for someone to argue that in 30-ish A.D. Judea, only men had the social freedom and capital to actually be a disciple, and that Jesus was simply a product of his times. And anything that calls the divinity of Christ into question is a non-starter. Can't even touch it, just to deny it.
Of course, there's more fringe-y stuff that got culled by the Council of Nicea, the Gnostic Gospels and others that had Mary Magdalene as a disciple, or possibly even Jesus' wife. And that has the exact same problem, if not worse, because it's a double-whammy. Unwanted questions if maybe "Jesus was just like, this guy... ya know?" and/or that if which Gospels were included and which weren't, or what role women took in them was based in local biases of the time, it calls the divine inspiration of the authors into question as well.
I don't want to give the impression of Christian bashing here. Because I'm not. And there's no point, because faith and logical arguments don't intersect, and you just butt heads from two completely different paradigms.
I certainly appreciate Christianity's role in Western Civ, and view it as an "ally" from a historical perspective, and as one of cultural practice, even if I can't from a spiritual or religious one. And those who seem hell-bent on actively tearing it down "just because" irks me to no end. When they could just not participate and live their own lives so easily. And these others who don't want to follow the rules of the club, and then insist they're still in the club... that's irritating too.