I can't help but wonder if this is a case of lost context with respect to this song since most people are familiar with it from countless covers and not the original version in context of the movie it was in, Neptune's Daughter (not a Christmas movie, but it was a hit, as was the song). The movie has two couples singing the song, switching the parts between the men and women, and its clear for both couples the concern is what people will say, not in their not wanting to stay and canoodle.
Even more ironic is that "Baby It's Cold Outside" was a substitute song according to Esther Williams. The Hayes office wouldn't approve "(I'd Like to Get You on a) Slow Boat to China" (supposedly they interpreted the "get" in a particular way). When "Baby It's Cold Outside" was nominated for an Oscar, other songwriters protested as Frank Loesser had created the song for him and his wife to sing at parties, but as it'd never been professionally performed, the MPAAS allowed its nomination to continue.
That's the issue, more or less. I never took the song to have the vibe people upset about it claim it has, always saw it as some elaborate dance. But oh well, this is Canada, where they had banned Money for Nothing due to one line