How can I say that? Consider the trouble caused by the usage of the word "niggardly", which has no racial connotations at all, and yet people who have used it have been given reprimands, been sued, or even lost their jobs for using a word that sounds similar to an unacceptable word. In fact it's now pretty much disappeared from the language altogether.
In the early days of feminism there was an effort to add some gender balance to language by changing words that had "man" in them to something a bit more gender-neutral. Some did take; for example "chairman" is now often "chair-person", or the slightly sillier (to my mind) "chair". Many did not. What I've noticed lately is something of a feminist revenge, where rather than attempting to "fix" the established language and persuade other people (particularly men) to use it, women have just started making up new male-specific words that often describe aspects of the female experience (eg. mansplain, man flu) which somehow hadn't been covered before.
And what's even more amusing, to me at least, is the rise of male-specific words to describe things that are traditionally more female associated, and why it's not in any way effeminate or un-manly for a guy to be associated with them (eg. man purse). Possibly used sarcastically, but your mileage may vary.
Not to be confused with words where someone has taken an offensive female-specific word and applied it to a man (eg man-slut). Equality of offensiveness is still just offensive.
A stroll through Urban dictionary has pages of this stuff, including many hilarious examples I've never heard of before, but am now noting down and looking for opportunities to insert them into conversation.