Also most animals judge by color, smell, and size, in their mating practices.
As far as color goes, you're a bit off. Color mostly comes into play in regards to catching attention, and aren't really an overall determining factor. For example, a male peacock has an assortment of vibrant colors that attract females' attention, amongst an assortment of other behavioral displays. More vibrant colors will draw the eye more and may result in the female paying closer attention to one male over the other. Ultimately, it's the song and dance routine that wins the female over. Having more vibrant coloration simply increases the chances that the female may not notice the other guy's song and dance in the first place. All male peacocks are the same color, after all.
In certain primate species, including humans, reddish hues may aid successful copulation for female members. During ovulation the genital region typically becomes engorged with blood, causing a reddish blushing hue that can signal fertility to males. In these species of primate, the males have typically been since evolved to be sexually drawn to the color red for this reason. Human females evolved to conceal signs of ovulation, as a means of promoting long term monogamous commitments from males (due to the unusually long gestational and maturation period of human young), but this trait is witnessed in human behavior, particularly the female practice of applying red lipstick (creating a pseudo-vagina displaying availability for copulation) and other makeup, and a strong frequency of male attraction to females with naturally red highlighting characteristics (i.e. red hair and freckles). Once again, this is not an attraction to the woman's skin color, but to a hue that overlays her skin color, and is simply an evolutionary indication of ovulation, nothing more.