There's no denying that a majority of the "Romance is icky" reaction is largely rooted in misogyny. I'll even go so far as to say that it's also largely rooted in homophobia. Here's why.
Recently, one of my dojo-mates (thanks Sheilagh!) sent out a link to a video of men reacting to sexual harassment. It's one of the best examples I've ever seen demonstrating that sexual harassment isn't about attractiveness or a 'compliment.' It's about power. It's about showing who has the power to conquer and removing all doubt as to who will be submissive to that conquest. It's similar to territory marking. Watch:
As you can see, the participants are often narrowly escaping having the crap beat out of them. And this is an instance where only hands are being touched. (Women endure much more intimate touches almost every day in my experience.) When I watched it, I was terrified in spite of the jovial music and the laugh track. Considering the video more closely, I can't think of a better analogy for what it's like to be in a situation that is both dangerous and that no one outside of it considers to be harmful. Most of the men I know react to this video with nervous laughter. Even the site my friend originally linked to does this. If that's your reaction, I suggest thinking closely as to why.
Now. Add to this T. Frohawk's blog post titled Women Write Romance, Men Write Manly Things. It's well worth the read. I'm going to pull a quote from it. "Take any sex scene in "Game of Thrones" and put it up against the wedding night scene in "Outlander." "Game of Thrones" is one hundred percent male gaze with the camera focused on the objectification of the female body whereas in the "Outlander" scene, the camera is focused entirely on Jamie, AND with a heavy emphasis on Claire leading Jamie through the act." The take-away here is male gaze versus female gaze. Thou shalt not sexualize men. Male gaze is the default setting--to borrow John Scalzi's phrase. Female gaze is not only non-standard, it's considered dangerous and/or disgusting. I used to think that the reason women's sexuality was so threatening to the patriarchy was because it justified/demonstrated women having the power of their bodies. I don't think that any more. Because when it comes down to misogyny, the core of it is not about the bigot's target. It's about the misogynist. Misogyny is motivated by a self-centered drive. Male is the default setting. Therefore, this fear of women's sexuality and/or the female gaze is probably more rooted in homophobia than anything else.
Now, let's think about these two stories together. What do they say? Why is it that in literature rape of a male character is considered extremely edgy and shocking while rape of a female character is an ordinary event? Why is it that Romance, a genre in which the female gaze is dominant, is so reviled? Why is it that this is considered so awful, so terrible, so horrific that all writers who happen to be women are instantly categorized as Romance Writers for fear of there being the possibility of exposure to female gaze? Before you answer, look up the psychological definition of generalization. Now. Think about all the above.
The answer is homophobia. Homophobia and misogyny are tightly linked, after all.
 This is complicated by the idea that some 'territory' is more valued than others. Once women are not of child bearing age, they are considered less 'valuable.' This doesn't mean the harassment stops entirely. It doesn't. This is a situation where a woman has something that a man wants/feels he has a right to have. That something can be a promotion and/or perceived power in a social group. That's why I'm telling you it's about power.
 In my experience, to the point that any piece in which such a scene appears must be firmly justified. So much so, that the piece may not even get published.
is a Science Fiction and Fantasy author living in Texas.