BluePhantom
Educator (of liberals)
As the general election heats up between Romney and Obama I am seeing a lot of posts pointing to this poll and that poll and I am noticing an equal amount of misunderstanding from all sides. As I am an admitted "poll geek" (to the point that I have a spreadsheet I wrote to analyze polling data - I know "get a life") I want to take some time to explain how to read polls and get the most from the information they offer. While I imagine there are threads on this topic from 2008 or 2010 it appears it's time for a refresher course at the very least.
Rule #1: Consider the Sample
You will generally see polls sampled in three ways. The first is "adults" (A). Polls that sample "adults" is the least reliable because only about 50%-55% of the eligible population actually turn out to vote. Polls of A might give you an idea of public perception but they don't tell you a lot about who is in the best position to win an election.
The second (and most common) is "registered voters" (RV). This is better than A polls because they disregard anyone who is not eligible and in a position to vote. Still they are not the best because only about 70% of registered voters actually go and cast a ballot. So it's better, but still slightly problematic. Quinnipiac and Gallup are examples of firms that use RV sampling.
The third (and best) is "likely voters" (LV). This considers only people who are registered to vote and meet a statistical criteria that indicates they actually will go vote. LV polls, with only a few rare exceptions (like Quinnipiac, for example) are the ones to pay the most attention to. Rasmussen and SurveyUSA are examples of firms that use a LV sampling method.
Rule #2. Understand Margin of Error
I see people all the time get so excited about a poll that shows their candidate up by 3%. In reality, from a statistical perspective that's a tie. Every poll will have a slightly different margin of error but a good rule of thumb is 4%. If a poll shows a lead of 4% or less, it's a statistical tie and could go either way.
Rule #3. Pay Attention to Timing
A poll in April about an election in November doesn't mean a whole lot. Too much can happen. The economy could dramatically recover or totally tank between those times. A scandal may break. We could get attacked and forced into a military confrontation. A candidate may get a temporary bounce from their party's convention, the selection of a running mate, or even a human interest story that captures the nation's attention. All of these things will influence the polls and voter preference. The closer to the election, the more valuable a poll becomes. This is why we experience the "October Surprise" (the dirty secret that a candidate exposes about their opponent a week or two before the election). Knowing what is happening and when can help you identify the difference between a trend that is likely to stick and a temporary bounce.
Rule #4. Know the Polling Agencies Affiliations and History
Any agency can luck out in a given year. It's important to know which firms show a history of accuracy over multiple election cycles. For example, the Washington Post was great on a few selected state polls in 2010. In 2008 however they were absolutely dreadful. Gallup has a great reputation but over the last several years they have been getting less and less accurate. Rasmussen had a surprisingly weak 2010 but in 2008 and for years prior they were absolutely deadly accurate. What changes? Sometimes their methodology, sometimes nothing....they had a bad year or a good year.
Also keep in mind that some firms are affiliated with a given party. Public Polling Policy (PPP), for example, is funded and affiliated exclusively with the DNC. Magellan Strategies, the RNC. Usually, on RealClearPolitics, thoss agencies are noted (D) or (R) for Democratic affiliated firms or Republican affiliated firms respectively. It's wise to keep in mind who is paying their bills when you consider the validity of their data. That's not to say these firms should be completely disregarded...just that it should be kept in mind.
Rule #5. READ THE FUCKING CROSSTABS
The crosstabs are information about the specifics of the polling demographics in that sample. They are usually at the very beginning or the very end of a polling report. Many liberals might be excited as hell with a poll that shows Obama with an 11% lead until they look more closely and notice that (simply by sheer chance) the polling agency reached a sample where 47% of them identified as Democrats compared to only 23% that identified themselves as Republicans. This creates what is known as the dreaded "outlier". Simply by sheer random chance the agency reached a given demographic that is out of proportion with the United States as a whole and it skewed the results to the point where the data is unreliable.
Rule #6. Trends and Averages are More Important Than Snapshots
A poll is basically a snap shot: "at this precise moment in time and according to the sample we reached, this is what the feeling is". The best way to read polls is to look at a collection of reliable polls and average the results. RealClearPolitics does this with the "RCP average" but that average does not consider all the points I have discussed. If it's a recent poll it gets counted whether it's a good poll or a bad poll.
Tracking the trends associated with the averages shows more than just what the snapshot is but where there is momentum toward one side or the other. It's the trends that matter more when election day is distant. Those snapshots only have real relevance a week or so away from election day because things can happen so fast that even a historically accurate poll can show a dramatic change in their data within a very short period of time depending on what happens to be going on at one point in time compared to the other.
So with all that said let me list in order the common agencies that, through my research and tracking on my spreadsheet, are the most valuable and the most accurate.
The Deadly Accurate Duo (First Tier)
1. SurveyUSA
2. Quinnipiac
Damned Accurate (Second Tier)
3. Rasmussen (slipped from First Tier after a shaky 2010)
Pay Close Attention To (Third Tier)
4. Mason-Dixon
5. PPP
Worth Consideration (Fourth Tier)
6. Gallup
7. Magellan Strategies
8. Strategic Vision
Consider With Care
9. ABC/Washington Post (one good year in a history of disaster does not establish confidence)
10. Fox News (historically getting more and more accurate but not there quite yet)
Best to Ignore
Pretty much everything else
By keeping these above points in mind a true "student of the polls" will be able to get a much more solid understanding of who is winning and losing, how they are winning, why they are winning, and will be able to distinguish between what is important and what is irrelevant. The ability to effectively analyze the polling data can also mean the difference between making a strong argument on a thread or being exposed as a complete tool. These concepts are vital to understand whether you are simply looking for ammo in a debate or you really want a true understanding of the political landscape and your candidate's chances for victory.
Rule #1: Consider the Sample
You will generally see polls sampled in three ways. The first is "adults" (A). Polls that sample "adults" is the least reliable because only about 50%-55% of the eligible population actually turn out to vote. Polls of A might give you an idea of public perception but they don't tell you a lot about who is in the best position to win an election.
The second (and most common) is "registered voters" (RV). This is better than A polls because they disregard anyone who is not eligible and in a position to vote. Still they are not the best because only about 70% of registered voters actually go and cast a ballot. So it's better, but still slightly problematic. Quinnipiac and Gallup are examples of firms that use RV sampling.
The third (and best) is "likely voters" (LV). This considers only people who are registered to vote and meet a statistical criteria that indicates they actually will go vote. LV polls, with only a few rare exceptions (like Quinnipiac, for example) are the ones to pay the most attention to. Rasmussen and SurveyUSA are examples of firms that use a LV sampling method.
Rule #2. Understand Margin of Error
I see people all the time get so excited about a poll that shows their candidate up by 3%. In reality, from a statistical perspective that's a tie. Every poll will have a slightly different margin of error but a good rule of thumb is 4%. If a poll shows a lead of 4% or less, it's a statistical tie and could go either way.
Rule #3. Pay Attention to Timing
A poll in April about an election in November doesn't mean a whole lot. Too much can happen. The economy could dramatically recover or totally tank between those times. A scandal may break. We could get attacked and forced into a military confrontation. A candidate may get a temporary bounce from their party's convention, the selection of a running mate, or even a human interest story that captures the nation's attention. All of these things will influence the polls and voter preference. The closer to the election, the more valuable a poll becomes. This is why we experience the "October Surprise" (the dirty secret that a candidate exposes about their opponent a week or two before the election). Knowing what is happening and when can help you identify the difference between a trend that is likely to stick and a temporary bounce.
Rule #4. Know the Polling Agencies Affiliations and History
Any agency can luck out in a given year. It's important to know which firms show a history of accuracy over multiple election cycles. For example, the Washington Post was great on a few selected state polls in 2010. In 2008 however they were absolutely dreadful. Gallup has a great reputation but over the last several years they have been getting less and less accurate. Rasmussen had a surprisingly weak 2010 but in 2008 and for years prior they were absolutely deadly accurate. What changes? Sometimes their methodology, sometimes nothing....they had a bad year or a good year.
Also keep in mind that some firms are affiliated with a given party. Public Polling Policy (PPP), for example, is funded and affiliated exclusively with the DNC. Magellan Strategies, the RNC. Usually, on RealClearPolitics, thoss agencies are noted (D) or (R) for Democratic affiliated firms or Republican affiliated firms respectively. It's wise to keep in mind who is paying their bills when you consider the validity of their data. That's not to say these firms should be completely disregarded...just that it should be kept in mind.
Rule #5. READ THE FUCKING CROSSTABS
The crosstabs are information about the specifics of the polling demographics in that sample. They are usually at the very beginning or the very end of a polling report. Many liberals might be excited as hell with a poll that shows Obama with an 11% lead until they look more closely and notice that (simply by sheer chance) the polling agency reached a sample where 47% of them identified as Democrats compared to only 23% that identified themselves as Republicans. This creates what is known as the dreaded "outlier". Simply by sheer random chance the agency reached a given demographic that is out of proportion with the United States as a whole and it skewed the results to the point where the data is unreliable.
Rule #6. Trends and Averages are More Important Than Snapshots
A poll is basically a snap shot: "at this precise moment in time and according to the sample we reached, this is what the feeling is". The best way to read polls is to look at a collection of reliable polls and average the results. RealClearPolitics does this with the "RCP average" but that average does not consider all the points I have discussed. If it's a recent poll it gets counted whether it's a good poll or a bad poll.
Tracking the trends associated with the averages shows more than just what the snapshot is but where there is momentum toward one side or the other. It's the trends that matter more when election day is distant. Those snapshots only have real relevance a week or so away from election day because things can happen so fast that even a historically accurate poll can show a dramatic change in their data within a very short period of time depending on what happens to be going on at one point in time compared to the other.
So with all that said let me list in order the common agencies that, through my research and tracking on my spreadsheet, are the most valuable and the most accurate.
The Deadly Accurate Duo (First Tier)
1. SurveyUSA
2. Quinnipiac
Damned Accurate (Second Tier)
3. Rasmussen (slipped from First Tier after a shaky 2010)
Pay Close Attention To (Third Tier)
4. Mason-Dixon
5. PPP
Worth Consideration (Fourth Tier)
6. Gallup
7. Magellan Strategies
8. Strategic Vision
Consider With Care
9. ABC/Washington Post (one good year in a history of disaster does not establish confidence)
10. Fox News (historically getting more and more accurate but not there quite yet)
Best to Ignore
Pretty much everything else
By keeping these above points in mind a true "student of the polls" will be able to get a much more solid understanding of who is winning and losing, how they are winning, why they are winning, and will be able to distinguish between what is important and what is irrelevant. The ability to effectively analyze the polling data can also mean the difference between making a strong argument on a thread or being exposed as a complete tool. These concepts are vital to understand whether you are simply looking for ammo in a debate or you really want a true understanding of the political landscape and your candidate's chances for victory.
Last edited:
But I love statistics so I crunch the numbers and the spreadsheet I wrote to do that crunching has proven to be very accurate I am proud to say. 
