DavidS
Anti-Tea Party Member
FiveThirtyEight.com: Electoral Projections Done Right
Tracking Poll Primer
Now that there are fully eight distinct national tracking polls, I thought I'd take the time to give you my Cliff's Notes assessment of each one. The polls are arranged in the order in which they typically appear throughout the day -- as you may have discovered, you can get your tracking poll fix quite literally morning, noon and night.
Zogby/C-SPAN/Reuters
When Publishes: About 12:45 AM Eastern Time.
Key Specifications: ~1200 likely voters, 3-day rolling sample.
Track Record: Zogby's telephone polls are not quite as bad a reputed, ranking just slightly below average (his internet-based Zogby Interactive polls are another matter altogether). However, they seem to be getting worse rather than better. Zogby was one of the few pollsters to call the popular vote for Gore in 2000, but he missed high on Kerry's numbers in many state-level polls in 2004, and then had an erratic primary season this year.
House Effect/Lean: For the reasons specified below, so far has leaned Republican by a point or two.
Features/Strengths: Zogby's write-ups are generally entertaining, and he reveals more than most about the progress of the candidates in individual days' results. One of two pollsters to publish results with a decimal place intact, which we like.
Zogby is the first pollster to go to press each day, publishing in the wee hours of the morning, although the way that he accomplishes this is to split his sample periods over two days. That is, a "day's" worth of polling consists of interviews from that afternoon plus the previous night.
Quirks/Concerns: There is one very, very significant concern with Zogby, which is that he has a longstanding rule to set his party weightings based on the exit polls from the most recent election. In this case, that means 2004, when a roughly equal number of Democrats and Republicans turned out. However, according to essentially every available poll, Democrats now have somewhere between a 5-point and a 10-point advantage in party ID.
This particular procedure has bitten Zogby in the ass before. Between 2000 and 2004, there was a shift in party ID toward the Republicans; as a result, Zogby's numbers were 2-3 points high for Kerry across battleground states.
To the extent that Obama is leading in a Zogby poll, that means essentially that he'd have won the election given 2004 turnout dynamics. 2008 turnout dynamics are liable to be sigificantly more favorable to him. With that said, the relationship between Zogby's odd party ID weightings and his toplines is probably not quite 1:1, since he is weighting based not just on party ID but also a number of other factors. My guess is, for instance, that one reason that Obama always seems to do strongly in Zogby among independents is that Zogby is essentially squeezing Obama-friendly demographic groups out of his Democratic pile and erroneously classifying them as indies.
Research 2000 / Daily Kos
When Publishes: Early AM, usually about 7:30 Eastern time.
Key Specifications: ~1100 likely voters, 3-day rolling sample.
Track Record: Research 2000 has an above-average track record, although this is the first time they've run a national tracking poll.
House Effect/Lean: Aggregating all of Research 2000 polls in this general election cycle -- including their tracking results plus their state polling for both Daily Kos and the myriad newspapers that they contract with -- they have had a slight Democratic lean of 1-2 points. However, the lean has appeared to be stronger in their national tracker, and less strong (in fact, almost nonexistent) in their state polling.
Features/Strengths: Full set of cross-tabular results published each day. Poll tends to be fairly stable, in part because they use a fixed party ID weighting (Democrat +9). Research 2000 is the only pollster to publish each individual day's results in addition to the rolling average.
Quirks/Concerns: Racial demographics are aggressive -- probably too aggressive -- showing blacks making up 14 percent of the electorate and Hispanics another 13 percent. Turnout will be up among minorities this year, but probably not by quite that much. The +9 party ID split is arguably also aggressive, although within the broad range of what other polls have found this year.
Rasmussen Reports
When Publishes: Promptly at 9:30 AM Eastern. Results are sometimes available slightly earlier for subscribers to their Premium service.
Key Specifications: 3000 likely voters, 3-day rolling sample.
Track Record: Rasmussen rates as a strong pollster overall and did particularly well in 2004, though less well in this year's primaries.
House Effect/Lean: Frequently reputed to have a Republican lean, Rasmussen's overall house effect between its state and national polls has in fact been very minor -- less than one full point. However, it has been somewhat more pronounced in their national tracker than in their state-level results.
Features/Strengths: Largest sample size of any of the tracking polls. Between that and the fact that they weight by party ID, they have tended to have the most stable results. To the extent that any pollster should weight by party ID, I think Rasmussen is going about it the right way, setting targets based on a six-week rolling average of all interviews conducted.
Quirks/Concerns: The poll sometimes crosses the line from being stable into lower-case-c conservative, as the strict party ID weighting scheme means that it often won't react as strongly as others to changes in the mood of the electorate. For the traditionalists out there, Rasmussen is the only one of the eight trackers to use the IVR/"robocall" method.
The Diageo Hotline Poll
When Publishes: Midmorning, usually around 10:30 AM Eastern time. Results are often teased on the Hotline Blog before they appear in full.
Key Specifications: ~800 likely voters, 3-day rolling sample.
Track Record: Not much information to go on. This poll was published in 2000 and called the race as Bush +3, but did not appear to be active in 2004.
House Effect/Lean: No pronounced house effects so far.
Features/Strengths: Only pollster to probe on enthusiasm each day in addition to the topline numbers. The National Journal folks take their politics very seriously and this poll is likely a reflection of that.
Quirks/Concerns: Has the smallest sample size of any of the trackers, and does not use a fixed party ID weighting. As such, their numbers have tended to bounce around a lot. That plus the limited track record make this a middle-of-the-road tracker.
Battleground / Tarrance / George Washington U.
When Publishes: Weekdays only -- midmornings around 10:30 AM Eastern time.
Key Specifications: Either 800 or 1,000 likely voters, spanning either 4 or 5 days respectively.
Track Record: Battleground is usually in hibernation, emerging only once every four years to produce national trial heat numbers. They've been around the block, however, and had great years in 1992 and 1996, followed by OK years in 2000 and 2004.
House Effect/Lean: Prior to a methodological change that they made a couple of weeks ago, had a very prounced (3-4 point) Republican lean. Since then, they've been fairly neutral.
Features/Strengths: Co-operated by a Republican (Ed Goaes) and Democratic (Celinda Lake) pollster, so to the extent they have a house effect, it isn't the result of partisan bias. Battleground publishes more cross-tabular detail than any other tracking poll, although the formatting is a bit cumbersome.
Quirks/Concerns: They don't conduct interviews on Fridays or Saturdays, which is a defensible decision, but means that the poll may be slower to react to new information early in the work week, especially when coupled with the relatively large (4-5 day) sample window.
As alluded to above, Battleground previously had significant problems with their sample, substantially undercounting young voters, which was likely causing the strong Republican lean. They have since revisited their turnout assumptions and altered their model, and have had more "normal" looking results since.
Gallup
When Publishes: At 1 PM Eastern time, or a few minutes thereafter.
Key Specifications: Generally about ~2750 registered voters and ~2200 likely voters over a rolling three-day window.
Track Record: Gallup's results lag a bit behind its reputation, as it ranks in the lower half of our accuracy ratings. Their likely voter models have been the subject of much discussion and occasional critique.
House Effect/Lean: Very neutral overall so far this year.
Features/Strengths: Large sample size. Includes a supplement of cellphone numbers. Write-ups are informative and concise. Strong professional ethics and sense of transparency. Significant cross-tabular information is available, although it's published on a weekly rather than daily basis.
Quirks/Concerns: The poll has been a bit "jumpy", reacting more strongly than others (i.e. perhaps overreacting) to events in the news cycle. Still, the main point of concern is with the three different versions that they publish each day (registered voters, and then two separate likely voter models) -- it's hard to know which version they are really throwing their weight behind. Also, the differences between the registered voter numbers and the more conservative version of their likely voter numbers are sometimes implausibly large. We advocate paying the most attention to the less conservative, so-called "Likely Voters II" turnout model, which is the number we list in our daily write-ups.
Investors Business' Daily / TIPP
When Publishes: Mid-afternoon, usually about 3 PM Eastern time.
Key Specifications: About ~1050 likely voters over a 5-day window.
Track Record: IBD/TIPP touts itself as the most accurate pollster based on its strong result in 2004, when they nailed the Bush-Kerry numbers within a couple of tenths of a point. One good result does not a pollster make, however, and in 2000, their performance was only average, missing the Bush-Gore margin by 2.5 points. Results were slightly erratic and counterintuitive earlier this year, when for instance they showed an 11-point Obama lead in mid-May when most other pollsters showed him struggling at that time.
House Effect/Lean: Not yet enough data to reach any firm conclusion.
Features/Strengths: Pretty good internals available at the IBD website. Publishes with the decimal place included.
Quirks/Concerns: A 5-day tracking window is relatively long, and means that IBD/TIPP may be slower to react to new trends. Their D-R party ID gap has been narrower so far than that of most other pollsters (perhaps too narrow), though it does not appear as though they weight based on party ID as Rasmussen or Research 2000 does. Overall, the poll is probably fine given these caveats, but we have relatively little basis on which to evaluate it.
ABC News / Washington Post
When Publishes: Embargoed until 5 PM Eastern.
Key Specifications: Just debuted yesterday, so it is hard to say with certainty, but they appear to be using a 4-day rolling sample of about 1300 likely voters.
Track Record: Strong relative to that of other major news organizations, which generally do not do terrific polling, although just average compared to the field at large. ABC/Post had a fairly strong go of things when they conducted polling during this year's primaries.
House Effect/Lean: Significant Democratic lean of 3 or so points so far this election.
Features/Strengths: The only tracking poll apart from Gallup to include a cellphone sample. Relatively large sample size. Results are documented and reported professionally.
Quirks/Concerns: I don't have any specific critiques about their methodology, although any poll with a house effect this large needs to be evaluated with that in mind.
*-*
In summation, none of these tracking polls are perfect, although Rasmussen -- with its large sample size and high pollster rating -- would probably be the one I'd want with me on a desert island. Conversely, the only one of the trackers that I consider obviously dubious is Zogby.
Tracking Poll Primer
Now that there are fully eight distinct national tracking polls, I thought I'd take the time to give you my Cliff's Notes assessment of each one. The polls are arranged in the order in which they typically appear throughout the day -- as you may have discovered, you can get your tracking poll fix quite literally morning, noon and night.
Zogby/C-SPAN/Reuters
When Publishes: About 12:45 AM Eastern Time.
Key Specifications: ~1200 likely voters, 3-day rolling sample.
Track Record: Zogby's telephone polls are not quite as bad a reputed, ranking just slightly below average (his internet-based Zogby Interactive polls are another matter altogether). However, they seem to be getting worse rather than better. Zogby was one of the few pollsters to call the popular vote for Gore in 2000, but he missed high on Kerry's numbers in many state-level polls in 2004, and then had an erratic primary season this year.
House Effect/Lean: For the reasons specified below, so far has leaned Republican by a point or two.
Features/Strengths: Zogby's write-ups are generally entertaining, and he reveals more than most about the progress of the candidates in individual days' results. One of two pollsters to publish results with a decimal place intact, which we like.
Zogby is the first pollster to go to press each day, publishing in the wee hours of the morning, although the way that he accomplishes this is to split his sample periods over two days. That is, a "day's" worth of polling consists of interviews from that afternoon plus the previous night.
Quirks/Concerns: There is one very, very significant concern with Zogby, which is that he has a longstanding rule to set his party weightings based on the exit polls from the most recent election. In this case, that means 2004, when a roughly equal number of Democrats and Republicans turned out. However, according to essentially every available poll, Democrats now have somewhere between a 5-point and a 10-point advantage in party ID.
This particular procedure has bitten Zogby in the ass before. Between 2000 and 2004, there was a shift in party ID toward the Republicans; as a result, Zogby's numbers were 2-3 points high for Kerry across battleground states.
To the extent that Obama is leading in a Zogby poll, that means essentially that he'd have won the election given 2004 turnout dynamics. 2008 turnout dynamics are liable to be sigificantly more favorable to him. With that said, the relationship between Zogby's odd party ID weightings and his toplines is probably not quite 1:1, since he is weighting based not just on party ID but also a number of other factors. My guess is, for instance, that one reason that Obama always seems to do strongly in Zogby among independents is that Zogby is essentially squeezing Obama-friendly demographic groups out of his Democratic pile and erroneously classifying them as indies.
Research 2000 / Daily Kos
When Publishes: Early AM, usually about 7:30 Eastern time.
Key Specifications: ~1100 likely voters, 3-day rolling sample.
Track Record: Research 2000 has an above-average track record, although this is the first time they've run a national tracking poll.
House Effect/Lean: Aggregating all of Research 2000 polls in this general election cycle -- including their tracking results plus their state polling for both Daily Kos and the myriad newspapers that they contract with -- they have had a slight Democratic lean of 1-2 points. However, the lean has appeared to be stronger in their national tracker, and less strong (in fact, almost nonexistent) in their state polling.
Features/Strengths: Full set of cross-tabular results published each day. Poll tends to be fairly stable, in part because they use a fixed party ID weighting (Democrat +9). Research 2000 is the only pollster to publish each individual day's results in addition to the rolling average.
Quirks/Concerns: Racial demographics are aggressive -- probably too aggressive -- showing blacks making up 14 percent of the electorate and Hispanics another 13 percent. Turnout will be up among minorities this year, but probably not by quite that much. The +9 party ID split is arguably also aggressive, although within the broad range of what other polls have found this year.
Rasmussen Reports
When Publishes: Promptly at 9:30 AM Eastern. Results are sometimes available slightly earlier for subscribers to their Premium service.
Key Specifications: 3000 likely voters, 3-day rolling sample.
Track Record: Rasmussen rates as a strong pollster overall and did particularly well in 2004, though less well in this year's primaries.
House Effect/Lean: Frequently reputed to have a Republican lean, Rasmussen's overall house effect between its state and national polls has in fact been very minor -- less than one full point. However, it has been somewhat more pronounced in their national tracker than in their state-level results.
Features/Strengths: Largest sample size of any of the tracking polls. Between that and the fact that they weight by party ID, they have tended to have the most stable results. To the extent that any pollster should weight by party ID, I think Rasmussen is going about it the right way, setting targets based on a six-week rolling average of all interviews conducted.
Quirks/Concerns: The poll sometimes crosses the line from being stable into lower-case-c conservative, as the strict party ID weighting scheme means that it often won't react as strongly as others to changes in the mood of the electorate. For the traditionalists out there, Rasmussen is the only one of the eight trackers to use the IVR/"robocall" method.
The Diageo Hotline Poll
When Publishes: Midmorning, usually around 10:30 AM Eastern time. Results are often teased on the Hotline Blog before they appear in full.
Key Specifications: ~800 likely voters, 3-day rolling sample.
Track Record: Not much information to go on. This poll was published in 2000 and called the race as Bush +3, but did not appear to be active in 2004.
House Effect/Lean: No pronounced house effects so far.
Features/Strengths: Only pollster to probe on enthusiasm each day in addition to the topline numbers. The National Journal folks take their politics very seriously and this poll is likely a reflection of that.
Quirks/Concerns: Has the smallest sample size of any of the trackers, and does not use a fixed party ID weighting. As such, their numbers have tended to bounce around a lot. That plus the limited track record make this a middle-of-the-road tracker.
Battleground / Tarrance / George Washington U.
When Publishes: Weekdays only -- midmornings around 10:30 AM Eastern time.
Key Specifications: Either 800 or 1,000 likely voters, spanning either 4 or 5 days respectively.
Track Record: Battleground is usually in hibernation, emerging only once every four years to produce national trial heat numbers. They've been around the block, however, and had great years in 1992 and 1996, followed by OK years in 2000 and 2004.
House Effect/Lean: Prior to a methodological change that they made a couple of weeks ago, had a very prounced (3-4 point) Republican lean. Since then, they've been fairly neutral.
Features/Strengths: Co-operated by a Republican (Ed Goaes) and Democratic (Celinda Lake) pollster, so to the extent they have a house effect, it isn't the result of partisan bias. Battleground publishes more cross-tabular detail than any other tracking poll, although the formatting is a bit cumbersome.
Quirks/Concerns: They don't conduct interviews on Fridays or Saturdays, which is a defensible decision, but means that the poll may be slower to react to new information early in the work week, especially when coupled with the relatively large (4-5 day) sample window.
As alluded to above, Battleground previously had significant problems with their sample, substantially undercounting young voters, which was likely causing the strong Republican lean. They have since revisited their turnout assumptions and altered their model, and have had more "normal" looking results since.
Gallup
When Publishes: At 1 PM Eastern time, or a few minutes thereafter.
Key Specifications: Generally about ~2750 registered voters and ~2200 likely voters over a rolling three-day window.
Track Record: Gallup's results lag a bit behind its reputation, as it ranks in the lower half of our accuracy ratings. Their likely voter models have been the subject of much discussion and occasional critique.
House Effect/Lean: Very neutral overall so far this year.
Features/Strengths: Large sample size. Includes a supplement of cellphone numbers. Write-ups are informative and concise. Strong professional ethics and sense of transparency. Significant cross-tabular information is available, although it's published on a weekly rather than daily basis.
Quirks/Concerns: The poll has been a bit "jumpy", reacting more strongly than others (i.e. perhaps overreacting) to events in the news cycle. Still, the main point of concern is with the three different versions that they publish each day (registered voters, and then two separate likely voter models) -- it's hard to know which version they are really throwing their weight behind. Also, the differences between the registered voter numbers and the more conservative version of their likely voter numbers are sometimes implausibly large. We advocate paying the most attention to the less conservative, so-called "Likely Voters II" turnout model, which is the number we list in our daily write-ups.
Investors Business' Daily / TIPP
When Publishes: Mid-afternoon, usually about 3 PM Eastern time.
Key Specifications: About ~1050 likely voters over a 5-day window.
Track Record: IBD/TIPP touts itself as the most accurate pollster based on its strong result in 2004, when they nailed the Bush-Kerry numbers within a couple of tenths of a point. One good result does not a pollster make, however, and in 2000, their performance was only average, missing the Bush-Gore margin by 2.5 points. Results were slightly erratic and counterintuitive earlier this year, when for instance they showed an 11-point Obama lead in mid-May when most other pollsters showed him struggling at that time.
House Effect/Lean: Not yet enough data to reach any firm conclusion.
Features/Strengths: Pretty good internals available at the IBD website. Publishes with the decimal place included.
Quirks/Concerns: A 5-day tracking window is relatively long, and means that IBD/TIPP may be slower to react to new trends. Their D-R party ID gap has been narrower so far than that of most other pollsters (perhaps too narrow), though it does not appear as though they weight based on party ID as Rasmussen or Research 2000 does. Overall, the poll is probably fine given these caveats, but we have relatively little basis on which to evaluate it.
ABC News / Washington Post
When Publishes: Embargoed until 5 PM Eastern.
Key Specifications: Just debuted yesterday, so it is hard to say with certainty, but they appear to be using a 4-day rolling sample of about 1300 likely voters.
Track Record: Strong relative to that of other major news organizations, which generally do not do terrific polling, although just average compared to the field at large. ABC/Post had a fairly strong go of things when they conducted polling during this year's primaries.
House Effect/Lean: Significant Democratic lean of 3 or so points so far this election.
Features/Strengths: The only tracking poll apart from Gallup to include a cellphone sample. Relatively large sample size. Results are documented and reported professionally.
Quirks/Concerns: I don't have any specific critiques about their methodology, although any poll with a house effect this large needs to be evaluated with that in mind.
*-*
In summation, none of these tracking polls are perfect, although Rasmussen -- with its large sample size and high pollster rating -- would probably be the one I'd want with me on a desert island. Conversely, the only one of the trackers that I consider obviously dubious is Zogby.
