Furthermore, traps fit into that sexual gray area that futanari also goes in. They don't really belong to either gender because of their extraordinarily feminine appearance and the fact they have a ♥♥♥♥. You can't call liking traps straight, because they're definitely not female, but you can't call it gay because they're really not all that male either.
To close it out, I understand the people who think it's gay. It's a reasonable stand-point. However, as someone who likes traps but firmly hates non-trap ♥♥♥♥♥, I have to disagree.

The main point is that people who like traps, as far as I can tell, aren't attracted to the fact they are male, but the fact they are a literal trap. Because it's not the "maleness" that attracts them, it isn't gay. I think the best example of this "genderless" attribute would be the iconic Hideyoshi, who is even referred to in-universe as some mystical third gender. This same idea carries over to many other traps, including the infamous reverse-trap, AKA the "You thought I was a guy acting like a girl, but really I was a girl acting like a guy acting like a girl."

The appeal of traps is the fact that they are so feminine, without actually being female. The feeling of the Bait-N-Switch it presents is a good one, the idea that a male is cuter than any girl is both extremely erotic and very, very adorable. It's not the fact that they're male and feminine, it's the fact that they aren't what they seem. It's the same reason many people like tomboys.
A slightly deeper meaning is that traps serve as this ideal that you aren't bound by gender roles or what people might think one should act like. Traps show that people can come in any attitude or personality and still give people a raging erection and drop panties.