- Thread starter
- Banned
- #41
Cherry-picking one survey to discredit a survey of scientists on climate change
Posted by Glenn Kessler at 06:00 AM ET, 05/08/2013
The Washington Post
Scientists are struggling to explain a slowdown in climate change that has exposed gaps in their understanding and defies a rise in global greenhouse gas emissions. (MICK TSIKAS/REUTERS - REUTERS)
"There is also uncertainty regarding to what degree man is to blame for global warming. However, the claim that 98 percent of scientists agree that humans are the singular driver of climate change has been repeatedly discounted. This oft-cited statistic is based on an online survey with a sample size of only 77 people, and the survey didn't even ask to what degree humans contribute to climate change." - Rep. Chris Stewart (R-Utah), chairman of the House Subcommittee on Environment, in an opinion article, April 13, 2013
Stewart is a freshman lawmaker who ended up with a plum position: heading a House panel on the environment. In an opinion article for the Salt Lake Tribune, he struck a cautious stance on climate change, arguing that the science is "anything but settled."
He, for instance, cited an interesting Economist article that the climate may be heating up less quickly in response to greenhouse gas emissions than previously thought. (He did not mention that the article also said "that does not mean the problem is going away.") For the purposes of this fact check, we will look at his claim about the 98 percent statistic, which he says "has been repeatedly discounted" and is based just on a survey of 77 people. What's he talking about?
The Facts
Stewart is referring to a survey done for the American Geophysical Union in 2009 by researchers for the University of Illinois in Chicago. Peter Doran, associate professor of earth and environmental sciences, along with former graduate student Maggie Kendall Zimmerman, in 2008 sent a simple survey with nine questions to more than 10,000 experts listed in the 2007 edition of the American Geological Institute's directory of geoscience departments.
They ended up getting responses from 3,146 scientists, and then publicized the results from two questions: (1) Have mean global temperatures risen compared to pre-1800s levels? (2) Has human activity been a significant factor in changing mean global temperatures? The results? About 90 percent of the scientists agreed with the first question and 82 percent with the second.
So where does the 98 percent statistic come from? That's from a subsample of the survey - climate scientists. The survey actually says the result is 97 percent, but Stewart is correct that it represented just a small group of people - 77 out of 79 people. Generally, with this sample size, one can expect a margin of error of about plus or minus 11 percentage points, or a range of 86 to 100 percent. That's still a pretty big margin.
Note that Stewart simply said "scientists" - not climate scientists. That makes a difference, as experts who study the climate appear much more convinced that human activity has affected the climate. That's shown by the survey, which found that less than half of petroleum geologists agreed with the second statement.
But Stewart ignored the main result from the survey of more than 3,000 people - that 82 percent of those surveyed believed human activity was a significant factor in higher global temperatures. That result has a margin of error of plus or minus 4 percentage points, and so it's a pretty strong majority.
Still, there have been concerns raised about the study. As Stewart noted, it was an opt-in online survey, and The Fact Checker has raised caution flags in the past about opt-in online polls . You generally cannot draw broad conclusions about such surveys. Critics have also focused on the wording of the questions in the survey, especially the second one. The question did not define the meaning of "significant," and one scientist might interpret the phrase differently from another. Professor Murray Goot of Macquarie University in Australia explored some of these issues in a 2011 paper, and he concluded that while questions could be raised about the phrasing of the questions and the methodology of the survey, the results were not significantly different from other surveys of scientists. Over time, such surveys have shown that increasingly large percentages of scientists believed that anthropogenic (human-generated) greenhouse gas emissions affected climate change. He concluded:
Concerns about the sampling and the number of respondents used in the Doran and Zimmerman study are secondary even when they are not misplaced. The key issue has to do with the wording of the questions. While the questions asked in Bray and von Storch [another survey of scientists] aren't exactly equivalent to the question 'Do you believe that anthropogenic greenhouse gas emissions were the primary factor (50% or more) in the observed mean global temperature increase since the mid-20'th century?' they are sufficiently close - and the distribution of responses are sufficiently clear - to suggest that the results reported by Doran and Zimmerman are unlikely, in any very material sense, to be misleading. Doran, in an e-mail, said that Stewart's reference to the survey in the opinion article was "very misleading." He said that the 79 climate scientists were "undeniable experts" - who both list climate science as their area of expertise and have published more than 50 percent of their recent peer-reviewed papers on the subject of climate change. He said 75 of 77 individuals, or 97.4 percent, agreed with the second statement; two of the 79 didn't answer this particular question.
Doran said that "'significant' to scientists has clear meaning. It refers to a meaningful relationship - not random. We thought carefully about the exact wording and worked closely with a survey expert to design the questions." He added: "There was no bias in picking these people. It was just whoever was listed in the directory which was the best source of contact information for earth scientists we could find. This in itself was no easy task. There is no electronic version of this database, so we cut the spline off the directory and scanned every page, ran the scanned pages through an optical character recognition program, and created a spreadsheet with names and emails that required a lot of cleaning of the raw scans to get to a viable mailing list. Of the >10,000 invitations to participate that went out, over 3,146 scientists responded to the survey. For an online survey of very busy people, this was a fantastic response."
Meanwhile, since 2009, there have been other surveys that also have mirrored the survey for the AGU. Rather than using a poll, another group of researchers in 2010 attempted to discern the opinion of climate researchers by examining their published work. This survey examined the published writings of about 1,000 scientists and concluded that 97 to 98 percent of the most actively published scientists had "striking agreement" with the primary conclusions of the Intergovernmental Panel on Climate Change (IPCC): that human-generated greenhouse gases have been responsible for "most"of the "unequivocal" warming of the Earth's average global temperature. Another approach was used by a different survey published in 2010, which asked more detailed questions than the Doran-Zimmerman study. This survey also found "significant agreement among scientists on nearly all elements of the climate change debate, except for a minority of ideologically conservative scientists who are less supportive of some policy choices, such as imposing taxes to discourage certain practices." There are clearly some scientists who do not support this consensus. Readers frequently e-mail The Fact Checker a list of 1,000 "international scientists" who have listed themselves as skeptics of man-made climate change. As the surveys listed above have demonstrated, there are certainly some scientists who are skeptical, even more so if they are not climate specialists. But such a self-selected survey does not provide a true picture of what most scientists believe in a rigorous and statistical manner.
Allison Barker, a spokeswoman for Stewart, thanked us for flagging the 2010 survey of climate researchers. "We will be sure to check that out, and the congressman can use that in future material," she said in an e-mail after we noted he seemed to be cherry-picking just one survey. We appreciate his staff's willingness to examine the issue in more depth.
The Pinocchio Test
Stewart is trying to stack the deck here. He focuses on one small aspect of the survey - the result of a relatively small sample of climate scientists - while ignoring the broader result that found a substantial majority of scientists believed that humans have made a significant contribution to climate change. Moreover, he ignores the many other surveys that have found results very similar to the survey he referenced, whatever its possible flaws. We wavered between Three and Four Pinocchios on this claim, but in the end settled on Four. Stewart, after all, is chairman of an environmental panel. He said the results of this survey have been "repeatedly discounted." He should know that beyond this single survey, there are many more that back up its findings on the broad support among scientists - particularly climate researchers - regarding the impact that humans have on climate change.
Posted by Glenn Kessler at 06:00 AM ET, 05/08/2013
The Washington Post
Scientists are struggling to explain a slowdown in climate change that has exposed gaps in their understanding and defies a rise in global greenhouse gas emissions. (MICK TSIKAS/REUTERS - REUTERS)
"There is also uncertainty regarding to what degree man is to blame for global warming. However, the claim that 98 percent of scientists agree that humans are the singular driver of climate change has been repeatedly discounted. This oft-cited statistic is based on an online survey with a sample size of only 77 people, and the survey didn't even ask to what degree humans contribute to climate change." - Rep. Chris Stewart (R-Utah), chairman of the House Subcommittee on Environment, in an opinion article, April 13, 2013
Stewart is a freshman lawmaker who ended up with a plum position: heading a House panel on the environment. In an opinion article for the Salt Lake Tribune, he struck a cautious stance on climate change, arguing that the science is "anything but settled."
He, for instance, cited an interesting Economist article that the climate may be heating up less quickly in response to greenhouse gas emissions than previously thought. (He did not mention that the article also said "that does not mean the problem is going away.") For the purposes of this fact check, we will look at his claim about the 98 percent statistic, which he says "has been repeatedly discounted" and is based just on a survey of 77 people. What's he talking about?
The Facts
Stewart is referring to a survey done for the American Geophysical Union in 2009 by researchers for the University of Illinois in Chicago. Peter Doran, associate professor of earth and environmental sciences, along with former graduate student Maggie Kendall Zimmerman, in 2008 sent a simple survey with nine questions to more than 10,000 experts listed in the 2007 edition of the American Geological Institute's directory of geoscience departments.
They ended up getting responses from 3,146 scientists, and then publicized the results from two questions: (1) Have mean global temperatures risen compared to pre-1800s levels? (2) Has human activity been a significant factor in changing mean global temperatures? The results? About 90 percent of the scientists agreed with the first question and 82 percent with the second.
So where does the 98 percent statistic come from? That's from a subsample of the survey - climate scientists. The survey actually says the result is 97 percent, but Stewart is correct that it represented just a small group of people - 77 out of 79 people. Generally, with this sample size, one can expect a margin of error of about plus or minus 11 percentage points, or a range of 86 to 100 percent. That's still a pretty big margin.
Note that Stewart simply said "scientists" - not climate scientists. That makes a difference, as experts who study the climate appear much more convinced that human activity has affected the climate. That's shown by the survey, which found that less than half of petroleum geologists agreed with the second statement.
But Stewart ignored the main result from the survey of more than 3,000 people - that 82 percent of those surveyed believed human activity was a significant factor in higher global temperatures. That result has a margin of error of plus or minus 4 percentage points, and so it's a pretty strong majority.
Still, there have been concerns raised about the study. As Stewart noted, it was an opt-in online survey, and The Fact Checker has raised caution flags in the past about opt-in online polls . You generally cannot draw broad conclusions about such surveys. Critics have also focused on the wording of the questions in the survey, especially the second one. The question did not define the meaning of "significant," and one scientist might interpret the phrase differently from another. Professor Murray Goot of Macquarie University in Australia explored some of these issues in a 2011 paper, and he concluded that while questions could be raised about the phrasing of the questions and the methodology of the survey, the results were not significantly different from other surveys of scientists. Over time, such surveys have shown that increasingly large percentages of scientists believed that anthropogenic (human-generated) greenhouse gas emissions affected climate change. He concluded:
Concerns about the sampling and the number of respondents used in the Doran and Zimmerman study are secondary even when they are not misplaced. The key issue has to do with the wording of the questions. While the questions asked in Bray and von Storch [another survey of scientists] aren't exactly equivalent to the question 'Do you believe that anthropogenic greenhouse gas emissions were the primary factor (50% or more) in the observed mean global temperature increase since the mid-20'th century?' they are sufficiently close - and the distribution of responses are sufficiently clear - to suggest that the results reported by Doran and Zimmerman are unlikely, in any very material sense, to be misleading. Doran, in an e-mail, said that Stewart's reference to the survey in the opinion article was "very misleading." He said that the 79 climate scientists were "undeniable experts" - who both list climate science as their area of expertise and have published more than 50 percent of their recent peer-reviewed papers on the subject of climate change. He said 75 of 77 individuals, or 97.4 percent, agreed with the second statement; two of the 79 didn't answer this particular question.
Doran said that "'significant' to scientists has clear meaning. It refers to a meaningful relationship - not random. We thought carefully about the exact wording and worked closely with a survey expert to design the questions." He added: "There was no bias in picking these people. It was just whoever was listed in the directory which was the best source of contact information for earth scientists we could find. This in itself was no easy task. There is no electronic version of this database, so we cut the spline off the directory and scanned every page, ran the scanned pages through an optical character recognition program, and created a spreadsheet with names and emails that required a lot of cleaning of the raw scans to get to a viable mailing list. Of the >10,000 invitations to participate that went out, over 3,146 scientists responded to the survey. For an online survey of very busy people, this was a fantastic response."
Meanwhile, since 2009, there have been other surveys that also have mirrored the survey for the AGU. Rather than using a poll, another group of researchers in 2010 attempted to discern the opinion of climate researchers by examining their published work. This survey examined the published writings of about 1,000 scientists and concluded that 97 to 98 percent of the most actively published scientists had "striking agreement" with the primary conclusions of the Intergovernmental Panel on Climate Change (IPCC): that human-generated greenhouse gases have been responsible for "most"of the "unequivocal" warming of the Earth's average global temperature. Another approach was used by a different survey published in 2010, which asked more detailed questions than the Doran-Zimmerman study. This survey also found "significant agreement among scientists on nearly all elements of the climate change debate, except for a minority of ideologically conservative scientists who are less supportive of some policy choices, such as imposing taxes to discourage certain practices." There are clearly some scientists who do not support this consensus. Readers frequently e-mail The Fact Checker a list of 1,000 "international scientists" who have listed themselves as skeptics of man-made climate change. As the surveys listed above have demonstrated, there are certainly some scientists who are skeptical, even more so if they are not climate specialists. But such a self-selected survey does not provide a true picture of what most scientists believe in a rigorous and statistical manner.
Allison Barker, a spokeswoman for Stewart, thanked us for flagging the 2010 survey of climate researchers. "We will be sure to check that out, and the congressman can use that in future material," she said in an e-mail after we noted he seemed to be cherry-picking just one survey. We appreciate his staff's willingness to examine the issue in more depth.
The Pinocchio Test
Stewart is trying to stack the deck here. He focuses on one small aspect of the survey - the result of a relatively small sample of climate scientists - while ignoring the broader result that found a substantial majority of scientists believed that humans have made a significant contribution to climate change. Moreover, he ignores the many other surveys that have found results very similar to the survey he referenced, whatever its possible flaws. We wavered between Three and Four Pinocchios on this claim, but in the end settled on Four. Stewart, after all, is chairman of an environmental panel. He said the results of this survey have been "repeatedly discounted." He should know that beyond this single survey, there are many more that back up its findings on the broad support among scientists - particularly climate researchers - regarding the impact that humans have on climate change.
Last edited:
