The color of a person's skin negates racist content in their words? That's an interesting theory.
When race baiting.... you bet. Think al shaprton.
Take the N word for example....
I don't think you understand the term racism.
rac·ism
   [rey-siz-uhm] Show IPA
noun
1.
a belief or doctrine that inherent differences among the various human races determine cultural or individual achievement, usually involving the idea that one's own race is superior and has the right to rule others.
2.
a policy, system of government, etc., based upon or fostering such a doctrine; discrimination.
3.
hatred or intolerance of another race or other races.
Racism is negatively stereotyping other people on the basis of race or displaying hate/intolerance of an individual or group on the basis of their race.
The n-word is a racial slur, but it isn't necessarily racist. Whether it is racist or not would depend on the context in which it is used.
It's kind of like the term "bitch."
I have a couple of friends who'll occasionally use the term in a joking way, directed a me, i.e., "bitch, please." That's very different from one of the little tiny-dicked men on this board referring to me as a bitch because I've whupped his ass in a debate.