That raised the question of whether a fill-in-the-bubble exam was really the best way to evaluate whether someone was suited to a leadership position in the fire department. In fact, the company that made the test admitted that some of the items were "irrelevant" in New Haven. One question, for example, asked the test-takers whether fire equipment should be parked "uptown, downtown or underground when arriving at a fire." The question was based on information relevant to New York City firefighters, and was on the exam even though the city of New Haven has no "uptown" or "downtown."
The fact that whites were disproportionately represented among the top scorers on the test is not surprising. Whites and some Asian American groups, on average, score higher than African Americans, Latinos and other Asian American groups on fill-in-the-bubble tests. Researchers have offered a host of reasons to explain racial gaps in test scores, including disparities in financial, educational and cultural resources, as well as psychological phenomena.
But New Haven did not scrap its promotion list simply because whites had higher scores on the test than minorities. The city understood that Title VII does not automatically prohibit employers from using tests on which whites do better than minorities. In fact, the law acknowledges that tests are useful for evaluating and comparing job applicants. But, under Title VII, it would be illegal for a city to promote firefighters based largely on a test that is not a good measure of a junior firefighter's worthiness to be promoted.
New Haven's attorney correctly interpreted Title VII to mean that the city's firefighter test should measure "who is going to be a good supervisor ultimately, not who is going to be a good test-taker." In other cases, judges have concluded, based on expert testimony, that written, multiple-choice tests for firefighter promotion like the one in this case contain the "fatal flaw" of failing to test for "supervisory ability." The company that made the New Haven Fire Department exam acknowledges that its test does not include any questions that measure a test-taker's ability to supervise or lead other firefighters in the line of duty.