No. You don’t.
If you have insight gained from history to conclude that a member of the KKK is a racist, that’s not even a contested conclusion. One thing on this topic about which we can agree.
Ditto if history has advised you that Skinheads and neo-Nazis are racist. Again, agree.
But here’s the rub: when someone says, in such obvious ways, that they are racist, it’s ok to take them at their word since history clearly demonstrates that KKK, Skinheads and neo-Nazis are unquestionably racist people.
However, when someone speaks about a political preference and doesn’t approve of those racist groups or racism, many still get labeled as “racists” on that fallacious basis. IOW, the term “racist” get misapplied by you (and by folks like IQ2) for nasty rhetorical purposes — whereas the people whom you accuse of racism may not harbor a racist belief anywhere in their hearts minds, souls, thinking or philosophy.