I grew up before desegregation had actually hit the burbs, and I was 4 before I saw a black person at the shopping mall and loudly commented to my mother, "Look, Mommy--that man drank too much chocolate milk!"
I remember her flustered apology to him and her hauling me off scolding to never say anything like that again. She never did actually explain why his skin was brown, though. I was just going through that phase where I believed if I swallowed an orange seed, a tree would grow out of my stomach.
So yeah, I guess it's racist.