Measuring Media Bias
It is very rare that an article in The Quarterly Journal of Economics finds its way into the mainstream media. But an article in its recent edition, by Tim Groseclose and Jeffrey Milyo, entitled "A Measure of Media Bias," did just that last week.
Most of the reports I read were based upon the press releases that announced the article's publication. Much of the details of the article, which is 46 pages in length, went largely unnoticed. Having just finished reading this piece, I thought I might comment upon it.
The authors theorize that the mainstream press is (a) more leftist than the average member of Congress and (b) more leftist than the average American voter. This is an argument that many have made in the past, but rarely has it been done systematically, i.e. with some kind of objective measure of bias. The authors of this article develop such a measure.
What they do, in essence, is compare the frequency with which members of Congress reference think tanks in House or Senate speeches to the frequency with which 20 news sources reference think tanks in their news reports. The theory behind this test is that a conservative member of Congress will probably mention the same think tanks that a conservative media outlet will mention - and a liberal member of Congress will probably mention the same think tanks that a liberal media outlet will mention. If you have an objective measure of liberal/conservative among members of Congress (and Groseclose and Milyo use each members' Americans for Democratic Action voting score - a standard measurement in academic work), you can then develop a measure of liberal/conservative for media outlets.
In their words:
As a simplified example, imagine that there were only two think tanks, and suppose that the New York Times cited the first think tank twice as often as the second. Our method asks: what is the estimated ADA score of a member of Congress who exhibits the same frequency (2:1) in his or her speeches? This is the score that our method would assign the New York Times.
As has been reported, they found that most think tanks are more liberal than the average member of Congress from the same period.
What has gone relatively under-reported are two interesting findings. First, the results of Groseclose and Milyo tend to go against the arguments made by people like Eric Alterman and Neal Gabler that the media has a corporate, and therefore conservative or pro-business, bias. One result that they find, for instance, is that News Night with Aaron Brown and Time magazine have different ADA scores, with the former being about 9 "points" more conservative than the latter. Yet, both are owned by Time-Warner. This does not amount to a strong refutation of the theory of "conservative bias" but nevertheless the result is problematic for those who advocate that idea.
Another point that has gone under-reported is that almost all media outlets are more liberal than the average Republican member of Congress and more conservative than the average liberal member of Congress. The media outlets that are measured would, if they were members of Congress, be mostly Democratic - but also mostly conservatively Democratic. So, it is important not to overstate the scale of bias they find. The direction of the bias is left, but the strength of the bias is not overwhelming.
One argument that Groseclose and Milyo make that I think is overstated is the argument that we now know that the media is more leftist than the average American voter. They make this argument by inferring what the average voter's ADA score is. They estimate the average member of Congress's ADA score -- weighting it to account for factors like gerrymandering, population variance among senatorial constituencies, and the absence of congressional representation for Washington DC -- and use this estimate as a proxy for the average American's ideological position on the ADA scale. The problem with this is that I think it gives too much credit to the average voter. The consensus among political scientists is that only about 30% of the American electorate is ideological in the sense that the ADA would rate members of Congress. Thus, comparing the ideology of the electorate to the ideology of the media is, by and large, like comparing apples to oranges. The public is largely, as Donald Kinder once put it, "innocent of ideology." Unfortunately, this article lacks a discussion of this compelling literature - and so their argument in this regard is not very persuasive. Furthermore, even if we assume that the public is ideological, we cannot assume that they vote for members of Congress based upon ideology. We know for a fact that they do not. Accordingly, it is difficult to infer the average voter's ideology based upon the average member's ideology.
A final point is worth mentioning. This article does not prove that the media is biased toward the left. It produces a result that is consistent with that theory. That is an important distinction. The fact that this article is in the Journal of Quarterly Economics indicates that the article is methodologically valid (i.e. all of their statistics are good) and theoretically plausible and honest (i.e. they are not putting forth some insane theory or some insignificant way of testing the theory).
But that does not necessarily mean that it is theoretically perfect. The objectivity of this measure of bias is, as mentioned earlier, legislative voting. But consider how this measure is "filtered" in the study. One moves from congressional vote choices, to the ADA's assessment of what constitutes liberal and conservative vote choices, to the ADA's assignment of liberal/conservative scores, to speeches by members where think tanks are referenced, to references in the media to think tanks, to ADA scores for the media, to a comparison between media and congressional scores. This is not a very parsimonious measure in that it takes many different steps across institutions (Congress, ADA, media) and across actions (vote casting, speechifying, news reporting). It might be possible to construct a more parsimonious indicator of media bias that is just as objective - and such an indicator might provide a different result.
(One such possible improvement might be an independent indicator of whether a think tank is liberal/conservative. Their theory assumes that liberals prefer to cite liberal think tanks. That is how they conclude that media outlets are liberal when they see them referencing the same think tanks. But they do not test that assumption. One would need an independent measure of think tank ideology, which they do not provide, to see whether liberals in fact use liberal think tanks, and vice-versa. Maybe they do not.)
It is also not necessarily the case that, simply because the media is biased in this domain (i.e. its use of think tanks in its reports), it is necessarily biased in all domains. For instance, the media might have a tendency to use pro-leftist think tanks, but it might show a conservative bias in its selection of news stories. Maybe, in other words, its bias is compartmentalized - in some instances it is leftist, in others it is rightist. This article is only directly measuring a type of bias. The fact that objective indicators of bias are difficult to develop, and so rarely done, means that we simply cannot say with certainty whether the media is generally biased.
In other words, the results they find are consistent with a general bias, but they do not "prove" that there is such a thing. It is very possible that these results could eventually be explained by an alternative theory that does not conceive of the media as being generally biased. That would first require the development of another objective indicator of media bias.
