You raise a good and fair point, and I'm not sure I have a good answer. There are some interesting cases where a white person has sued a university over a race conscious admissions program. The most recent case was Fisher v University of Texas. Equal Protection played a central role. Fisher lost because the court defended U.Texas' right to use racial admission standards to promote diversity. Why does diversity matter to a university? One reason is that a diverse student body gives students a first hand experience with other world views and life experiences, which - in the right environment - can be a valuable form of learning.
So yes, I think universities should have a right to select for diversity, but I also think the post-50's Left has overplayed race, gender and lifestyle to the point of alienating the traditional working class that FDR tried to help with the New Deal. Ideally, I'd want the Left to be responsive to all the losers of globalization and stagnant wages, regardless of color.
Okay, if you think universities have a right to use affirmative action, do you also think a baker or a photographer has the right to refuse to participate in a gay wedding?
Our Constitution protects religion and equality, not diversity. After all, if you tell a person they can't attend a college or get a job based on their white race, it's still discrimination even if you give it a fancy term like affirmative action. The only time equal protection comes to play is if you're not white.
That begs the question, will we ever have a country where everybody is considered equal, or will we always have special rules for special people constitution be damned?