Let me pick this apart a bit for you.
Yes it was fought mostly by white men, because blacks, on both sides, weren't ALLOWED to fight moron, that combine with the fact that A) Most northerners were white and B) the south had a higher black population than the north. In the north blacks weren't allowed to fight and in the south, well you'd be retarded to mass arm a group of people you have oppressed at the same time as having a larger, better equipped, and more powerful army attacking you at the same time, that was advocating their freedom. That's just simple. And when eventually blacks were allowed to fight, their units, per capita, performed better than their white counterparts.
This is a wonderful ideal to have now, however, de facto and de jure inequality of the past has led to inequities now. As a result, simply saying "O well we don't do that any more we're all equal" does not work because those who have already been disenfranchised are already at a disadvantage. That's what the purpose of affirmative action (esp for colleges) and other similar programs do.