The reason most White people don't speak out against "racism" is that extremely over-used, mis-used word means different things to different people. Clarify what is meant by "racism," then ask why Whites don't speak out against it.
I am an elderly White American.
What I believe the word, racism, means is hatred of any and all members of a given racial category for no reason other than their racial identity. I harbor no such orientation and I can recall having known only one individual whom I believe did feel that way toward the Negro race. I don't know how he felt toward Asians and/or other racial categories.
I have no doubt there were (are) more like this fellow but I'm sure they represent no more than a tiny percentage of the White population.