Polling Data May Be Bad for GOP or Not

[Annie]

I read the WaPo article yesterday, I thought pretty much the same thing. For me, Bush has been too liberal on domestic issues. Link to the WaPo article at site:


http://www.realclearpolitics.com/articles/2006/04/debunking_the_washington_posts.html

April 18, 2006
Debunking the Washington Post's Morin
By Jay Cost

Richard Morin, The Washington Post's polling director, had an interesting piece in yesterday's edition. Morin argues that, in states Bush carried in 2004, his job approval numbers have slid so much that we can no longer really think of many of them as "red".

Writes Morin, "States that were once reliably red are turning pink. Some are no longer red but a sort of powder blue. In fact, a solid majority of residents in states that President Bush carried in 2004 now disapprove of the job he is doing as president. Views of the GOP have also soured in those Republican red states." This conclusion is entirely unjustified by the evidence he cites - for four reasons.

The first two problems are different types of invalid causal inferences. Bush's job approval is down nationwide - of this there is no dispute. Morin, however, takes this as a sign that the GOP's electoral coalition is cracking up. In other words, according to Morin, Bush's job approval is a reliable indicator of the change in the ideological/partisan dispositions of the public. Is this valid? Can we infer that individual red voters are turning blue based upon the fact that Bush is losing ground among them? No way. It is true that it could signal this, but it is also just as likely to signal a whole host of other sentiments.

Consider a few alternatives. Conservatives might be angry with Bush for being too liberal on issues like spending and immigration. It is unlikely that they will move into the Democratic column. Moderates and independents might be upset because Bush now seems to be incompetent - but this does not imply a lasting increase in electoral power for Democrats. This is an evaluation of Bush's person that we cannot assume affects his party's electoral fortunes. These explanations are just as reasonable as Morin's. Many more are equally reasonable as well. Thus, to infer that it is the decline of conservatism or Republicanism is invalid.

The second problem has to do with his argument about the GOP's standing. Morin asserts that, in red state America, voters now trust Democrats more than Republicans to handle the most important issues of the day. He sees this as a sign of an electoral shift. However, take a close look at the history of this question (#9 found halfway down this page), noting the generally heavy and persistent Democratic skew that has been extant for almost 15 years. This is the case despite the fact that the public has increasingly trusted the GOP to run the government over the same period of time. This question thus seems to have much in common with the generic congressional ballot - both are poor indicators of the partisan sentiment in the nation because both tend to point toward the Democrats regardless of what actually happens. This question, therefore, is not a valid indicator of the future of our electoral alignment.

His third problem is what we might call an invalid descriptive inference. Morin asserts that the decline of Bush's job approval has hit deep red state America just as much as everywhere else. He shows that Bush is down to 47% approval in deep red territory. In a moment, we are going to call that statistic into question. But let us take it for granted for now. What does it mean? Does it mean that Bush is down evenly across the 25 states that he carried by 5% or more? Does it mean he is down in all of them? Of course not! He could be down more in some places than others. He could be up almost everywhere and down heavily in a few places. He could be down everywhere and up heavily in one place. You cannot tell from this figure alone.

This is known as the ecological fallacy - you run face first into it every time you try to draw inferences about parts of a whole based only upon data about the whole. Take an example. Suppose you want to determine how the giving pattern of the parishioners at your church has changed over the last year. You compare this year's statistic to last year's and you find that the amount is the same. Can you conclude that everybody is giving the same amount? No. A whole host of possibilities are valid based upon the data - everybody could be giving the same, but it is also possible that almost everybody is giving nothing and two people are making up the difference. It is the same case here. Is this a decline in all of red state America, a decline in half of it, a third, a tenth? You cannot tell from the aggregate data.

The fourth problem is also an invalid descriptive inference. It has to do with the error inherent to his statistics. When I started reading this article, I was very intrigued. I thought ABC News/Washington Post had commissioned some grand 50-state poll to see what Bush's job approval actually is across the states. It turns out, as I learned in the 11th paragraph (!), that Morin has merely divided his most recent national sample into four quadrants: strong Bush states, moderate Bush states, strong Kerry states, moderate Kerry states.

The problem with this is that it greatly increases the size of the sampling error. This should cut down on the number of conclusions Morin draws - but in practice it does not. He keeps on arguing as if a tiny sliver of a national poll is just as reliable as the whole poll.

If the ABC News/Washington Post poll perfectly reflects the nation's politico-geographical divisions, Morin's sample of strong Kerry states should be about 363 people; for strong Bush states, 449 people; for moderate Kerry states, 133 people; for moderate Bush states, 81 people.

It is about these moderate Bush states that he draws his most interesting conclusions - that a "solid majority" now disapprove of Bush. However, it is here that his margin of error is so high that his data is the most ambiguous. He does not give the exact proportion of Bush supporters in those states, but the margin of error for a sample of 81 people would be a little more than +/- 10%. Therefore, to validly infer that Bush is below 50% among the population of moderate Bush states, you would have to find his job approval to be 39% in the sample. Even if he did find a statistic that was statistically certain to indicate approval of less than 50%, the error rate would still be so high that it would not be nearly as indicative as he claims it is.

Morin also mentions that 47% of respondents from strong Bush states approve of the job he is doing. If the number of these people in the sample are proportional to their number nationwide, then Bush's job approval number in strong red areas could range from anywhere between 42% and 52%. In other words, the margin of error is +/- 5%, a very important figure to consider, given that Bush is 3% below the half mark! Thus, Morin cannot validly infer that a majority in these areas disapproves of Bush.

There is actually reason to suspect that Morin's sample of red America has too many people from shallow red states and too few people from deep red states. The overall job approval for Bush in all red states is 43%. In strong red states Morin finds it to be 47%. Either Bush is doing much worse in the shallow states than he is nationwide (specifically, a job approval between 18% and 24%!), or the sample draws too lightly from the deep red states. The latter is obviously more likely. This means that, so long as we assume that people in any given deep red state support Bush more than people in any given shallow red state, we must conclude that approval of Bush in all red states is higher than the 43% Morin finds. All in all, an under sample of the deep red states enhances the "sexiness" of Morin's column, but probably undermines the validity of its argument.

As I said, Morin does draw some reasonable inferences. Bush's job approval is down nationwide. The data is clear on this. It is also clear that it is down in these quadrants. Unfortunately, he also infers so much more than what is reasonable. He ignores the margins of error to declare that Bush is more unpopular than popular everywhere in America. He indicates, running afoul the ecological fallacy, that Bush is down systematically within these quadrants. He draws an invalid causal inference about why this is the case. He draws another invalid causal inference to claim that the whole GOP is in grave danger. This is the equivalent of pulling not one rabbit, but four rabbits, out of a hat. It is a nifty trick, but it is still a trick.

Remember, all of Morin's inferences could be true - but that would not change the fact that many of them are invalid. Even a broken clock is correct twice a day. I am not asserting that the state of the world is different from the one Morin sees. I am, rather, asserting that Morin does not, cannot, see this world from his data.