http://reviewipcc.interacademycounc...tion of IPCC’s Assessment Processes.pdf
Recommendation
The IPCC should make the process and criteria for selecting participants for scoping
meetings more transparent.
The IPCC has no formal process or
criteria for selecting authors, although some Working Group Co-chairs established their own for the fourth assessment, considering factors such as scientific expertise and excellence, geography, gender, age, viewpoint,and the ability to work in teams.9 Establishing such criteria and applying
them in a transparent manner to all Working Groups would alleviate some of the frustrations voiced.
An analysis of the 14,000 references cited in the Third
Assessment Report found that peer-reviewed journal articles comprised 84
percent of references in Working Group I, but comprised only 59 percent
of references in Working Group II and 36 percent of references in
Working Group III (Bjurström and Polk, 2010).
Non-peer-reviewed sources are to be listed in the reference sections of
IPCC reports, followed by a statement that they are not peer-reviewed. The
objectives are to ensure that all information used in IPCC reports receives
some sort of critical evaluation and its use is open and transparent, and
that all references used in the reports are easily accessible.
Although the Committee finds that IPCC’s procedures in this respect
are adequate, it is clear that these procedures are not always followed.
Some of the errors discovered in the Fourth Assessment Report had been
attributed to poor handling of unpublished or non-peer-reviewed sources
(Ravindranath, 2010). Moreover, a search through the Working Group
reports of the fourth assessment found few instances of information
flagged as unpublished or non-peer-reviewed
Handling the full range of views
An assessment is intended to arrive at a judgment of a topic, such as the
best estimate of changes in average global surface temperature over a specified
time frame and its impacts on the water cycle. Although all reasonable
points of view should be considered, they need not be given equal
weight or even described fully in an assessment report. Which alternativeviewpoints warrant mention is a matter of professional judgment
Equally important is combating confirmation bias—the tendency of
authors to place too much weight on their own views relative to other
views (Jonas et al., 2001). As pointed out to the Committee by a presenter10
and some questionnaire respondents, alternative views are not always
cited in a chapter if the Lead Authors do not agree with them. Getting the
balance right is an ongoing struggle. However, concrete steps could also be
taken. For example, chapters could include references to all papers that
were considered by the authoring team and describe the authors’ rationale
for arriving at their conclusions.