Annie
Diamond Member
- Nov 22, 2003
- 50,848
- 4,827
- 1,790
http://www.chicagotribune.com/news/...886.story?coll=chi-newsopinionperspective-hed
There's lot more to polls than just numbers
By not focusing on respondents' party affiliations, the news media are failing to fully explain the significance of opinion surveys
By Meg Kreikemeier, a stay-at-home mom and a former analyst at CNA Insurance
May 22, 2005
In early October 2004, Newsweek released a poll immediately after the first presidential debate, which showed a dramatic shift in public opinion in favor of John Kerry.
Did Kerry narrow the gap with his debate performance? Was he really the closer that many in the media had suggested he was? Newsweek was basing its headline "The Race is On" and accompanying story on a comparison between its two most recent polls. The problem, though, was that the polling data was inconsistent.
The October Newsweek poll sampled more Democrats than it did Republicans.
And the first poll, conducted in September 2004, sampled more Republicans than Democrats, not at all reflective of the historical composition of registered voters.
In fact, the Annenberg Public Policy Center released a study in November 2004 comparing party affiliation of registered voters.
The study showed that the gap between Democrats and Republicans had narrowed but that Democrats still outnumbered Republicans by about 3 percentage points.
Given the swing in demographics between the two Newsweek polls, of course Kerry saw improvement in his results. In fact, if he hadn't he would have been in deep trouble.
And while President Bush's support among Republicans eroded a bit between the polls, his support among Democrats actually increased. Kerry's support among Republicans went up slightly, and his support among Democrats remained flat.
So why the breathless headlines?
Why did the news media report the data without first thoroughly reviewing it?
Why did the change in poll results pique the curiosity of a stay-at-home mom like me but not the much-ballyhooed investigative instincts of the reporters covering the election?
Had Newsweek paused and considered the shift in party breakdowns within its polls, it might have had an interesting story to pursue.
Reporters could have questioned whether Bush received the typical bounce that each candidate hopes for from his convention. Were more Republicans willing to be interviewed or were more Americans actually considering themselves Republicans rather than Democrats, against historical trends?
Recall that, like Reagan Democrats, people do not necessarily alter their party affiliation when supporting a candidate from the opposing party.
Instead the headlines were written and the issue was framed without the underlying poll data being scrutinized. The race was always "on" and likely much closer than the headlines suggested. Damage was done to the candidate perceived to be losing momentum, first to Kerry when the number of Republicans outnumbered Democrats and then to Bush when the demographics shifted back to historical patterns.
Damage was done because most people believe what they read, trusting that the data had been vetted.
Now fast forward to the beginning of April.
Shortly after the pope's death, the Associated Press released a poll asserting that most Americans and Catholics wanted to see changes in the Catholic Church. But was that really true?
It would have been true if the headline had read "Most Liberals, Whether Catholic or Not, Want Changes in the Catholic Church" because that is what the data reveal. The breakdown of the data by party indicated that Democrats accounted for 51 percent of the poll's respondents while Republicans accounted for only 36 percent, hardly a representative breakdown of the population in general.
Further, Catholics only accounted for 23 percent of the poll's sample, which may be a fair representation of Catholics in the population, but why would non-Catholics play a larger role in a poll about the Catholic Church than Catholics themselves? And why, after all, would the opinion of non-Catholics matter?
Wouldn't their tendency be to represent their own beliefs because they don't have a vested interest in the Catholic Church anyway?
Only 89 of the approximately 1,000 people interviewed were both Republican and Catholic. If one assumes that they represent the views of traditional, conservative Catholics, then that group would not have any impact on the overall results, and that number may even be overstated because more than half of the Republicans surveyed considered themselves moderately as opposed to strongly Republican.
I'd wager, as I think most would, that conservative Catholics don't want any changes in the church and that liberal Catholics do.
The poll results were what they were because the AP questioned more liberals, both Catholic and non-Catholic. That's not news and certainly not worthy of a headline. But the poll made the headlines, and the issue was framed: Americans want changes in the Catholic Church.
Why did the press latch onto this poll as gospel?
I don't think many would argue with the view that the media as a group is left-of-center, and I contend that this poll went unquestioned because it fit the media's Democratic-leaning views. That is, the poll jibed with the media's own hypothesis that the Catholic Church should change and so the conclusion seemed right.
I believe this same thought process occurred with the release of the April 26 ABC/Washington Post poll about filibusters of judicial appointments and Social Security reform. To quote the Washington Post article, "As the Senate moves toward a major confrontation over judicial appointments, a strong majority of Americans oppose changing the rules to make it easier for Republican leaders to win confirmation of President Bush's court nominees. ... The wide-ranging survey also recorded a precipitous decline in support for the centerpiece of Bush's Social Security plan--private or personal accounts. ..."
This poll was used by the networks to help buttress Democrats' views that the Republicans shouldn't change the filibuster rules or Social Security. On the "Jim Lehrer News Hour," Margaret Warner cited the poll in a Social Security debate between Max Baucus and Chuck Grassley.
A quick perusal of the ABC/Washington Post poll data showed that Democrats outnumbered Republicans 35 percent to 28 percent, and while party identification does shift, the percentages are in the high range for Democrats and the low side for Republicans.
The 7-percentage-point difference between the two parties is also on the high end of what recent polls have found. Given that, moderation in the news reporting would have made sense. Interestingly, a Fox News poll released a day after the ABC/Washington Post poll showed a very different picture on Social Security while having the same job approval rating for President Bush.
But that poll got very little play, if any, in the news.
The Washington Post poll was quoted by MSNBC, CBS, as well as the Washington Post and ABC. Again, the headlines and news reports were written and the issue was framed as: Americans don't want changes to the filibuster rules or private accounts in Social Security.
One could say that I give the media more credit than it deserves for its knowledge of polls, but a claim of ignorance is not a valid defense. One could make the argument that the media rely on their polling firms to conduct fair polls, and therefore do not feel compelled to delve into the background data.
But a lack of diligence is a cop-out and hypocritical because the media don't allow the same excuse for those they investigate.
Transparency is crucial if the media plan to market polls as truth.
To give credit to the pollsters, many are now releasing their data as links to companion articles on the Internet. It's a step forward.
However, none of the demographic breakdowns or shifts in these polls were mentioned in the news reports, and I suspect the reason for that is because the media never questioned the data to begin with.
As a result, the opportunity for honest debate was lost.
