You're right. I've known several gay people over my years. Matter of fact on of places I lived, I rented a room from 2 lesbians, one of whom had a gay cousin. I'd talked to them quite a bit, and one of the things that they kept bringing up is that homophobes are always scared that the gays are going to bend them over and start having their way with them. Unfortunately for the phobes, most of them are not good looking and generally have lousy personalities, and women wouldn't look at them, so why should the gay men? I mean.....most gay men are better looking than most straight guys, because of the competition in their dating circles.
Personally, if you ask me, it's ego driven fear. They (the homophobes) think just because they have an anus they are attractive to gays, when in reality, they (the homophobes) aren't really attractive to anyone else, they're just looking for something to feed their ego.