What you deviates are confused with is the idea that it just isn't whether the fellow inflicted with this "feels" like a broad, its how they are perceived by others.
If a dude wants to dress up like a broad, he ain't fooling anyone.
Maybe the XX and XY chromosomes might not be "congruous" to how the possessor feels, but everyone else won't be so easily tricked on a social level.
On a political level, people may be able to toe the liberal line and pretend like the dude in the dress is a dame. But on a personal level, they won't be treated other than as the man they are.