You shouldn't. I'm just pointing out that how you personally deal with being called a cracker by a black man has nothing to do with his intensions for using the term. That's why it's not racist for a white person to call another white person, "cracker," just as it's not racist for a black person to call another black person, "******."
Using your (fucked up) logic a black man calling another black man an Uncle Tom isn't racist.
No, it is not. Perhaps you're just not aware of what the word, 'racism,' means ...
Racism - Definition and More from the Free Merriam-Webster Dictionary
poor treatment of or violence against people because of their race
the belief that some races of people are better than others
1 : a belief that race is the primary determinant of human traits and capacities and that racial differences produce an inherent superiority of a particular race
2 : racial prejudice or discrimination
A black person calling another black, an "uncle Tom," while it's certainly meant as an insult to highlight that individual's bending overly backwards for a white person, their intent for using the term does not fit that definition.