The Truth Behind the 97% Number

A mirrored posting from Quora: https://martinstoehr.quora.com/The-Truth-Behind-the-97-Number


Based on an answer I provided on Quora and a subsequent discussion (https://www.quora.com/Why-is-onl...), I finally decided to break down for others, my concern about bias from the original Cook et al study which is the source for the oft miss-quoted consensus number.

My deeper analysis started with a comment from Mr Tarr, and here is the statement that finally drove me to document my concerns:

"Cook et al conducted a scientifically rigorous analysis of published abstracts on climate science and reached a scientifically defensible conclusion that the abstracts showed a general consensus regarding AGW. I don’t know why you’d claim that’s not science."

Mr Tarr’s assumption was not much different than the others I have encountered on Quora and elsewhere.

For a link to the original Cook et al “letter” from which a number of quotes are taken directly: http://iopscience.iop.org/articl...

The following is my step-by-step analysis of the Cook study, why it is biased ‘research’, how it goes off the rails, and why people should not blindly believe there is an all powerful consensus backing up their opinions…

----------

To start, Cook's own assessment of the "type" of research the team was performing was not claimed to be rigorous science:

2. Methodology
This letter was conceived as a 'citizen science' project by volunteers contributing to the Skeptical Science website... we searched the ISI Web of Science for papers published from 1991–2011 using topic searches for 'global warming' or 'global climate change'.

As noted, these were"citizen scientists" doing a web search to gather a collection of papers which had "global warming" or "global climate change" listed as a topic in the ISI Web of Science database (Web Of Science).

The Cook et al methodology section goes on to describe how the initial team of 12 categorized and then rated the abstracts of about 11,530 of the 11,944 papers they found in their search. They worked in pairs and had a third reviewer join the rating effort if the initial two reviewers disagreed (which occurred 33% of the time).

Then I submit the team's own assessment of bias (emphasis is mine):

Two sources of rating bias can be cited: first, given that the raters themselves endorsed the scientific consensus on AGW, they may have been more likely to classify papers as sharing that endorsement. Second, scientific reticence ... or 'erring on the side of least drama' (ESLD...) may have exerted an opposite effect by biasing raters towards a 'no position' classification. These sources of bias were partially addressed by the use of multiple independent raters and by comparing abstract rating results to author self-ratings.

As noted, the Cook et al team readily admits they are biased. Unfortunately they do not provide any information on how they validated the unbiased perspective of the"multiple independent raters" nor did they indicate who these raters were or how many were involved. As for the "ESLD"—that is a term (and paper: Climate change prediction: Erring on the side of least drama?) arising from the Climate Change community itself and has very little credibility as a means to prove "an opposite effect [of bias]". Circular arguments are not usually permitted in a logical deduction and I see this self-ascribed anti-bias as little more than an excuse to be scientifically lazy.

With the above indications from the Cook et al team, I hold a high level of skepticism which AGW believers may not. As Mr Tarr had previous posted, Cook et al conducted a "scientifically rigorous analysis…" "and reached a scientifically defensible conclusion" ...and thus accused me (rightly) of "…suggesting that the scientists who found a 97% consensus WEREN’T QUALIFIED to conduct or interpret their study."

So let us see how qualified and biased these 'scientists' may have been. Here is a list of the lead authors to Cook et al and their qualifications:

• Cook - BS Physics -blogger, author, cartoonist 
• Nuccitelli - BS Astrophysics, MS Physics - researcher, author 
• Green - Environmental Chemist - faculty Michigan Tech U 
• Richardson - Physics,PhD Climate - NASA JPL developer 
• Winkler - ? - blogger, zoo volunteer 
• Painting - ? - police officer, environmentalist 
• Way - BA Geography -student, researcher 
• Jacobs - MS Environmental Science and Policy - student, researcher 
• Skuce - BSc Geology, MSc Geophysics - consultant, surveyor, author 

Note that every one of these people are associated with the SkepticalScience web site and blog (https://www.skepticalscience.com...). I would draw attention to the fact that 3 of 8 may not have the typical credentials needed to qualify them as 'scientists' let alone 'climate scientists' or 'experts'.

Here is the list of people who were claimed to have "collect[ed] email and rat[ed] abstracts":

• Jokimaki - BSc Computer Engineering - blogger, author * 
• Reitano - PhD Physics- materials research * 
• Honeycutt - ? -entrepreneur * 
• Cook - ? - ? 
• Scadden - BSc Geology- thermal modelling, geochemistry 
• Tamblyn - Mechanical Engineering - researcher, IT * 
• Blackburn - BSc Environmental Policy - blogger * 
• Hartz - ? - blogger 
• Brown - BSc Geosciences - IT security, student * 
• Morrison - ? - ? 
• Coulter - Earth Sys Science and Engineering - student 
• Stolpe - Climatology,Meteorology - researcher 

Note that 6 of 12 (marked with *) are also directly associated with the SkepticalScience web site and blog. In this list, 4 of 12 do not seem to be 'credentialed'.

I will not speak negatively on those who do not have a technical degree nor are directly educated in the field of 'climate science' because I have not ascribed a valid definition of such a thing and I do not care about authority, I simply care about the data and the science. I presume that anyone who took basic, university-level courses in the hard sciences (Physics, Chemistry, Biology) should be more than qualified to speak to these papers’ conclusions. I will leave it to the reader to decide if they are willing to accept the conclusions of bloggers, volunteers, environmentalists, entrepreneurs, and students—heretofore refer to as non-accredited—as credible researchers and 'scientists'.

Given that Cook et al used an initial pool of 12 reviewers with each abstract being initially assigned to two of those 12—I calculate that they could have had one of every 22 abstracts reviewed by two non-accredited individuals (3/12 x 2/11 = 0.0455 assuming 3 in 12 were non-accredited). Similarly about 1 out of 4 abstracts could have had at least one non-accredited reviewer.

Now, here is the trickery they used to get their 97% number…

First they categorized the abstracts of 11,944 papers with their own arbitrary system and “found” the following results:

We find that 66.4% of abstracts expressed no position on AGW, 32.6% endorsed AGW, 0.7% rejected AGW and 0.3% were uncertain about the cause of global warming.

(BTW, these are my graphs. The Cook team did not seem to be interested in providing data visually—maybe because it does not help their argument)

Then they took the sub-categorized abstracts and found that within those 3896 papers; 3783 of the original papers explicitly expressed that position (my emphasis) :

Among abstracts expressing a position on AGW, 97.1% endorsed the consensus position that humans are causing global warming.

There it is: 3783 papers of 11944 papers endorsed the consensus AGW and expressed such a position in the abstract—clear as you would expect from any scientific research paper, right?

Note that the final percentage of papers that “endorse the consensus” out of all the papers reviewed is 31.7% (3783/11944), not 97.1%.

Plus you need to remember, these results are all based on the opinions of the Cook et al team. What is amusing is that the ‘research team’ was able to compare their opinions to some actual authors and found that they were no better than chance at rating the abstracts. As noted in Table 5, based on direct responses from the authors (discussed below), half of the abstract ratings were mis-categorized—they would have saved more time by rolling dice.

----------

I argued with Mr Tarr that rather than using their own rating system, the Cook team could have conducted an actual direct poll of the researchers responsible for each of the abstracts / papers they assembled. As it turns out, to some extent they did exactly that (reference the Supplementary Information document,section S2):

S2. Survey of authors
Email addresses ... were determined for [8547] scientists... The text of the self-rating survey form provided to authors follows. 

... and in the report, these self-ratings were summarized thus:

... 2142 papers received self-ratings from 1189 authors. The self-rated levels of endorsement are shown in table 4. Among self-rated papers that stated a [Cook et al] position on AGW, 97.2% endorsed the consensus. Among self-rated papers not expressing a [Cook et al] position on AGW in the abstract, 53.8% were self-rated as endorsing the consensus.

Unfortunately for Cook's team the response rate from 1189 authors was not as impressive a number as the original 11,944 papers that they started with. Also unfortunate for the ‘science’ aspects of their citizen project, rather than reporting on the survey data they collected (ie which authors responded with what endorsements), the Cook 'research team' applied a number of biases to those direct-email poll results:

1. they did not provide the direct-poll data as a reference in their paper 
2. they summarily dismissed the mediocre 'consensus' response 
3. they arbitrarily re-categorized the direct-poll responses in order to gain a larger 'consensus' percentage 
4. they claim the trend was toward more endorsements of the AGW position yet their own data show it trending down over the time period of their investigation (Figures 1b and 2b) 

Table 4 shows this relevant information:

% of respondents [of the 2142 papers that:]
"Endorse AGW" = 62.7%
[hold] "No AGW position" = 34.9%
"Reject AGW" = 2.4%

Re-read those results because they come directly from the paper's authors—The consensus of scientists that endorse AGW from the direct poll is less than 63%!



Taking either their own rating system or the direct responses from authors, the 97% consensus number is completely indefensible and is essentially made-up by the Cook et al ‘citizen science’ team.

One could argue that the author response rate to the poll would be biased in favor of those who endorsed the AGW perspective. Because of the public reputation of the Cook et al team at the time and the language of the survey (noted in the S2 supplementary information section), this would not be a surprise. Even if it was not slanted, the absolute most Cook et al could claim would be a 63% endorsement, not 97%.


Unfortunately 63% endorsement does not make for sensational headlines, nor does it really indicate much of a consensus. So Cook et al, plays one more trick to get back to their original number. Again, by cherry-picking an arbitrary category of authors “among papers with [an] AGW position” they return to a 97.2% of authors “Endorse AGW”.


This just goes to prove:

"There are three kinds of lies: lies, damned lies, and statistics.
—Benjamin Disraeli (Lies, damned lies, and statistics - Wikipedia)

----------

I personally reject the sub-categorization of "% among respondents with AGW position” shown in Table 4. Remember, the self-admitted, biased authors of Cook et al were the ones who categorized papers as "endorsing AGW"; they cherry-picked a sub-category to strengthen their 'consensus' number. What difference does it make if an author's abstract was deemed to endorse AGW, as judged from the perspective of Cook’s team? Is it not more important that the authors STATE they endorse AGW?

What may have been more relevant was the qualification of the papers' authors as 'climate scientists'. Cook himself frequently uses that term throughout the propaganda which followed the public release of Cook et al—but no where in the paper do they provide backing for which scientists were ‘climate scientists’.

Remember, the “abstracts expressing a position on AGW” is their own sub-category and they chose it to be 97%! It could be whatever they wanted it to be. Cook's team does not even expressly state that 97% of 'climate scientists' endorse AGW, only that their team’s metric of "expressing a position on AGW" in the original abstract is sufficient.

So it is my opinion that the Cook et al team demonstrates and admits to being biased, that they misrepresent the poll numbers which they collected but did not share, AND they continue to spread misinformation through the media to the point that John Cook brags about President Obama's slanted tweet:

He also lead-authored the paper [...], which was tweeted by President Obama

President Obama's Tweet (my emphasis):

Ninety-seven percent of scientists agree: #climate change is real, man-made and dangerous. Read more:...

No Mr Obama, 97% of scientists did not agree that “climate change is real, man-made and dangerous”; but Cook et al wants you to believe they did.

Hopefully that explains my perspective on Cook et al's ‘citizen science’ project.