Even men who are very comfortable with a masculine identity resent being "Sirred" and served under their skin by subservient feminists.
I don't like to be called "Sir" or "Mister" but that does not mean I am militant or identify as a woman. It is because those are terms for old people and I prefer to think I am eternally 19
I occassionaly say yes ma'am to a man and have had it done to me by mistake. Have never freaked out about it. We are just culturally more accustomed to saying Yes, ma'am than we are yes, sir I think.