Ah, the good old false equivalency gag.
Not really. Infringing on someone's individual rights, when they haven't done so to someone else, isn't okay in any instance, regardless of their ideology, dress, mannerisms, belief, or any number of preferences, ideas, or individual qualities. It's inherently unethical.
Rape isn't "infringing on someone's individual rights|". Rape is a violent ciminal assault.
Calling someone out for wearing a symbol of racism, xenophobia, and the election slogan of a man who is an agent of the Russian government is the very essence of "free speech".
Every time one of you right wing assholes complains about other people violating their rights and freedoms, it turns out that what they're complaining about is other people's freedom of expression to comment on and disagree with their political messaging. If you're going to turn yourself into a walking billboard for hate, expect people to comment on it.