Poll Reading 101

[BluePhantom]

does someone here have the stasitcs of what % of registered voters are R/Conserv/Ind/and Dems? what we really need to know is how many registered voters are dems, then compare it to the rest.they say 40% of us are conservatives. so how are the other 60% caculated? is it true that millions of dems have switched red over the last 2 years?

I am not so sure that's a good way to measure it. Lots of people don't bother to change their registration even though they stopped voting with their party. When I lived in Phoenix I changed my registration to Democrat so I could vote in the primary between Clinton and Obama and simply never changed it back. Also party identification can change so quickly. I can guarantee you for example that on September 14, 2008 there was one set party identification and on September 16, 2008 it was totally different.

I would argue that the self-identification is probably more accurate than the registrations so long as you look at the most recent polling data available.

 

[California Girl]

Purely from an academic standpoint, I prefer polls that have a decent size poll. I tend to dismiss 910 as not a large enough sample to really tell me anything.

I tend to agree. The larger the sample size the more comfortable I am with the results.

It is something I'm quite anal about.... the amount of people who read the headline results and accept that as reality without bothering to interrogate the actual poll sometimes astonishes me. 612 people in New Jersey is not a solid premise for an outcome. (That's the one the left quote about Fox News viewers being less intelligent.... 612 people? And who are they calling stupid now? The irony is delicious!)

Also, as you pointed out earlier.... the wording of the question is absolutely vital to establish an outcome that is representative. Those of us with an academic mind know that to get an honest answer, you must ask an honest question.

 

[NYcarbineer]

Again.

How do you know a poll is accurate?

in what sense? How do you know they are releasing the actual data they collected? How do you know if it's an accurate representation of what they are trying to measure? Be more specific as to what you are referring to

The purpose of a poll is to take a sample and come up with an accurate representation of a larger population.

Unless you know where the entire population stands on a poll issue, you never have any way of knowing if a poll is accurate. Final election polls are about the only testable polls.

 

[NYcarbineer]

Phantom I just posted the most recent fox poll. What's do you take away from it?

Well the first thing I see is that it's a RV poll, which is good...not great but good. However, Independents are underrepresented. Party identification varies in the USA but as a general rule of thumb it's pretty much split into thirds between the Republicans, Democrats, and Independents. So on this poll the breakdown is:

Democrats (n = 361) ± 5%
Republicans (n = 354) ± 5%
Independents (n = 173) ± 7.5%

Now because Democrats and Republicans are reasonably even it doesn't create as much of a problem, but it is something to file away in the back of my mind. We can have a look at the crosstabs and extrapolate a bit of information.

For example in question #1 we have:

"1. Do you approve or disapprove of the job Barack Obama is doing as president?"​

Well the total is 42% approval vs 51% disapproval...BUT among Independents it's 37% - 53% respectively. So if we consider that Independents are underrepresented by about 50%, in reality Obama's approval rating is probably a couple points lower than the 42% the poll suggests.

.

Your reasoning is fatally flawed because you're assuming that the Independents sampled in the poll accurately represent Independents.

Independents can be anything from far left to far right. In a poll that oversampled conservative Independents, as this one may well have done, then approval of the president is going to skew against him,

which it did in this poll, and which is a good bet why this poll was an outlier.

Secondly, that R vs D ratio is laughably skewed in favor of Republicans.

The best indicator of political balance in a poll is liberal/conservative/moderate self-identification, not Republican/Democrat/Independent.

 

[Seawytch]

There is only one accurate poll and it will be held on the first Tuesday after the first Monday in November.

 

[NYcarbineer]

Nate Silver did extensive analysis of the pollsters in the 2010 election and found Quinnipiac the best and

Rasmussen the worst, with a huge bias towards Republicans:

http://fivethirtyeight.blogs.nytime...rate-quinnipiac-surveyusa-performed-strongly/

 

[nodoginnafight]

You're right Carbineer:
OP is obviously a rassmussen apologist (and a poor one at that)

 

[NYcarbineer]

You're right Carbineer:
OP is obviously a rassmussen apologist (and a poor one at that)

Quite an elaborate effort though.

 

[nodoginnafight]

You're right Carbineer:
OP is obviously a rassmussen apologist (and a poor one at that)

Quite an elaborate effort though.

It's pretty typical on these boards: More words = less intelligence

 

[BluePhantom]

Nate Silver did extensive analysis of the pollsters in the 2010 election and found Quinnipiac the best and

Rasmussen the worst, with a huge bias towards Republicans:

I wouldn't put a whole lot of faith in Nate's analyses. I used to frequent fivethirtyeight before the 2008 elections and pointed out several occasions where he cooked the numbers to favor the liberal position. This got much, much worse after he became affiliated with the New York Times and ultimately he...ahem..."invited me to stop pointing out the statistical flaws in his work or accept banishment". At this point he's really little more than a Democratic propagandist who attempts to use statistics to prove that people in Wyoming actually want to vote for Obama.

Silver's ranking of pollsters is very heavily biased. For example in his calculations he gave bonuses to groups who hold membership in the NCPP/AAPOR. Why? What does that have to do with the accuracy of their results? It has a lot to do with how transparent an agency is in releasing their methodology but simply because they don't choose to be a part of an organization doesn't mean they are suddenly inaccurate. But Silver penalizes those who are not in his calculations. Well Rasmussen doesn't do that because they have a service where poll geeks can pay for membership to see more detailed information. As a result Silver applies a penalty to Rasmussen. He does that with Mason-Dixon and PPP as well.

Now that being said, I pointed out that in the 2010 midterms, Rasmussen had a shaky showing compared to their usual standards and results. That's why I dropped them from a first tier organization to a second tier organization. But do you bench Drew Brees because he threw a pick six? Do you put in a pinch hitter for Babe Ruth because he struck out his last time at bat? Like I said in the OP....sometimes an agency will have a bad year and it's worth noting that Rasmussen does a vastly greater number of polls than anyone else. More than double the next most frequent pollster. That's going to impact their accuracy average as well. If you throw five passes in a game you will probably avoid being intercepted. If you throw 50 even the best quarterbacks in history will toss it to the other team a few times.

 

[nodoginnafight]

See what I mean

 

[NYcarbineer]

Nate Silver did extensive analysis of the pollsters in the 2010 election and found Quinnipiac the best and

Rasmussen the worst, with a huge bias towards Republicans:

I wouldn't put a whole lot of faith in Nate's analyses. I used to frequent fivethirtyeight before the 2008 elections and pointed out several occasions where he cooked the numbers to favor the liberal position. This got much, much worse after he became affiliated with the New York Times and ultimately he...ahem..."invited me to stop pointing out the statistical flaws in his work or accept banishment". At this point he's really little more than a Democratic propagandist who attempts to use statistics to prove that people in Wyoming actually want to vote for Obama.

Silver's ranking of pollsters is very heavily biased. For example in his calculations he gave bonuses to groups who hold membership in the NCPP/AAPOR. Why? What does that have to do with the accuracy of their results? It has a lot to do with how transparent an agency is in releasing their methodology but simply because they don't choose to be a part of an organization doesn't mean they are suddenly inaccurate. But Silver penalizes those who are not in his calculations. Well Rasmussen doesn't do that because they have a service where poll geeks can pay for membership to see more detailed information. As a result Silver applies a penalty to Rasmussen. He does that with Mason-Dixon and PPP as well.

Now that being said, I pointed out that in the 2010 midterms, Rasmussen had a shaky showing compared to their usual standards and results. That's why I dropped them from a first tier organization to a second tier organization. But do you bench Drew Brees because he threw a pick six? Do you put in a pinch hitter for Babe Ruth because he struck out his last time at bat? Like I said in the OP....sometimes an agency will have a bad year and it's worth noting that Rasmussen does a vastly greater number of polls than anyone else. More than double the next most frequent pollster. That's going to impact their accuracy average as well. If you throw five passes in a game you will probably avoid being intercepted. If you throw 50 even the best quarterbacks in history will toss it to the other team a few times.

Ah, so the guy who puts 'educator (of liberals)' under his username wants to shoot the messenger Nate Silver

on an unsubstantiated charge of partisan bias?

Did you REALLY think you could slip that one through? Don't insult me.

 

[BluePhantom]

The purpose of a poll is to take a sample and come up with an accurate representation of a larger population.

Unless you know where the entire population stands on a poll issue, you never have any way of knowing if a poll is accurate. Final election polls are about the only testable polls.

Yes that's true. I mentioned that in the OP. What the polls say now really doesn't mean a whole lot except to tell you where things appear to be now. The things to be looking for at this point is not so much the spreads, but the momentum and the trends. I would say that late September is when you might want to start taking the polls a bit more seriously with increasing importance through election day.

I held this argument with Candycorn on a different thread. She pointed out that the RCP average was Obama +4 and as such it was "over"; a "done deal". Well that's obviously ridiculous. The momentum will shift back and forth dramatically between now and then.

So yes I would agree with you that the only poll we know to be accurate is the one where people go and cast their ballot. But you can look at those organizations who have a history of being accurate and make a reasonable assumption that they reflect the current state of mind of the nation. I listed those agencies in the OP

 

[NYcarbineer]

The purpose of a poll is to take a sample and come up with an accurate representation of a larger population.

Unless you know where the entire population stands on a poll issue, you never have any way of knowing if a poll is accurate. Final election polls are about the only testable polls.

Yes that's true. I mentioned that in the OP. What the polls say now really doesn't mean a whole lot except to tell you where things appear to be now. The things to be looking for at this point is not so much the spreads, but the momentum and the trends. I would say that late September is when you might want to start taking the polls a bit more seriously with increasing importance through election day.

I held this argument with Candycorn on a different thread. She pointed out that the RCP average was Obama +4 and as such it was "over"; a "done deal". Well that's obviously ridiculous. The momentum will shift back and forth dramatically between now and then.

So yes I would agree with you that the only poll we know to be accurate is the one where people go and cast their ballot. But you can look at those organizations who have a history of being accurate and make a reasonable assumption that they reflect the current state of mind of the nation. I listed those agencies in the OP

But there's no need to look at individual pollsters if you agree that the averages are a better indicator.

 

[BluePhantom]

Ah, so the guy who puts 'educator (of liberals)' under his username wants to shoot the messenger Nate Silver

on an unsubstantiated charge of partisan bias?

Did you REALLY think you could slip that one through? Don't insult me.

If all Nate did was "deliver the message" I wouldn't shoot him. Unfortunately, that's not what he does. Consider:

In his rankings on June 6, 2010 Silver lists his rankings based on "PIE" (pollster introduced error) instead of "rawscore" which is historically how accurate they have been. He argues that regardless of past results the best way to predict their future dependability is through PIE. Problem....he never really explains how he distinguished between avoidable errors and unavoidable errors and he regresses the statistics toward different means. So if a firm is part of those organizations, they will be regressed toward one mean. if they are not, they will be regressed toward a completely different mean.

Now that is a decision not based upon results, historical accuracy, or anything related to the integrity of the data. It's based completely on Silver's personal determination of what is avoidable vs. unavoidable and when he cannot see their methodology (like with Rasmussen) he assumes the error is avoidable and regresses them toward a dramatically more unfavorable mean.

That my friend is not statistical analysis. That's politics and that's bias.

I encourage you to read Mark Blumenthal's excellent article regarding Nate's ratings for a more detailed explanation.

 

[nodoginnafight]

Silver is as good as there is in the business.
Folks who have a problem with Silver just have a problem with the truth

 

[BluePhantom]

But there's no need to look at individual pollsters if you agree that the averages are a better indicator.

I do agree that they are a better indicator. My suggestion is not so much that one poll should be treated as gospel above all others. But there are the reliable polls and then there are the really unreliable polls. Surely you would agree that the best indicator would be to identify which agencies are historically unreliable, disregard them, and consider the averages of those agencies which do show historical accuracy.

If I have a set of four polls for example (lets say Rasmussen, Quinnipiac, Gallup, and TIPPonline). Well I am not going to include the TIPP poll because it's unreliable, we know it's unreliable, and it's not wise to introduce unreliable data into an analysis. Neither will I say "well I have Quinnipiac ranked higher so I will just accept their findings". No I will average the remaining three out and say "reality probably lies within 3-4 points of the result at this moment in time."

 

[BluePhantom]

Silver is as good as there is in the business.
Folks who have a problem with Silver just have a problem with the truth

And I am sure Nate and the NY Times are thrilled that you let them do your thinking for you.

 

[BluePhantom]

Your reasoning is fatally flawed because you're assuming that the Independents sampled in the poll accurately represent Independents.

Independents can be anything from far left to far right. In a poll that oversampled conservative Independents, as this one may well have done, then approval of the president is going to skew against him,

which it did in this poll, and which is a good bet why this poll was an outlier.

Secondly, that R vs D ratio is laughably skewed in favor of Republicans.

Not really.

According to Gallup and Rasmussen the split between party identification is as follows

Party.......Gallup........Rasmussen
Dem.........30%.............33.4%
Rep..........27%.............36.4%
Ind...........42%.............30.2%

Now historically Rasmussen is a bit more accurate than Gallup but just for simplicity let's average the two out with equal weight and get a more likely breakdown of:

Dem: 31.7%
Rep: 31.7%
Ind: 36.1%

This would be more in line with historical trends. Now look at the Fox Poll sample

Dem: 361 or 40.7%
Rep: 354 or 39.9%
Ind: 173 or 19.5%

The balance between Democrats and Republicans is less than 1% which is right about where it should be according to the averages of the Gallup and Rasmussen reports. But the number of Independents is over 16 points lower than it should be according to the same data. So it's not that Republicans are over-represented any more than Democrats are and frankly the Democrats are slightly more over-represented than the Republicans. It's that Independents are underrepresented.

The best indicator of political balance in a poll is liberal/conservative/moderate self-identification, not Republican/Democrat/Independent.

Yeah I would tend to agree with you on that point. Like I said, there are a lot of things about this particular poll I find a little concerning and the sample was one of them. As far as liberal/conservative/moderate self-identification vs Republican/Democrat/Independent, I think that's a fair point.

 

[NYcarbineer]

Your reasoning is fatally flawed because you're assuming that the Independents sampled in the poll accurately represent Independents.

Independents can be anything from far left to far right. In a poll that oversampled conservative Independents, as this one may well have done, then approval of the president is going to skew against him,

which it did in this poll, and which is a good bet why this poll was an outlier.

Secondly, that R vs D ratio is laughably skewed in favor of Republicans.

Not really.

According to Gallup and Rasmussen the split between party identification is as follows

Party.......Gallup........Rasmussen
Dem.........30%.............33.4%
Rep..........27%.............36.4%
Ind...........42%.............30.2%

Now historically Rasmussen is a bit more accurate than Gallup but just for simplicity let's average the two out with equal weight and get a more likely breakdown of:

Dem: 31.7%
Rep: 31.7%
Ind: 36.1%

This would be more in line with historical trends. Now look at the Fox Poll sample

Dem: 361 or 40.7%
Rep: 354 or 39.9%
Ind: 173 or 19.5%

The balance between Democrats and Republicans is less than 1% which is right about where it should be according to the averages of the Gallup and Rasmussen reports. But the number of Independents is over 16 points lower than it should be according to the same data. So it's not that Republicans are over-represented any more than Democrats are and frankly the Democrats are slightly more over-represented than the Republicans. It's that Independents are underrepresented.

The best indicator of political balance in a poll is liberal/conservative/moderate self-identification, not Republican/Democrat/Independent.

Yeah I would tend to agree with you on that point. Like I said, there are a lot of things about this particular poll I find a little concerning and the sample was one of them. As far as liberal/conservative/moderate self-identification vs Republican/Democrat/Independent, I think that's a fair point.

Prove that. Show me that you know the real numbers of total party breakdown, historically, and then show me that Rasmussen was/is closer than any other pollster.