The CRU hack: Context
- Climate Science
This is a continuation of the last thread which is getting a little unwieldy. The emails cover a 13 year period in which many things happened, and very few people are up to speed on some of the long-buried issues. So to save some time, I’ve pulled a few bits out of the comment thread that shed some light on some of the context which is missing in some of the discussion of various emails.
- Trenberth: You need to read his recent paper on quantifying the current changes in the Earth’s energy budget to realise why he is concerned about our inability currently to track small year-to-year variations in the radiative fluxes.
- Wigley: The concern with sea surface temperatures in the 1940s stems from the paper by Thompson et al (2007) which identified a spurious discontinuity in ocean temperatures. The impact of this has not yet been fully corrected for in the HadSST data set, but people still want to assess what impact it might have on any work that used the original data.
- Climate Research and peer-review: You should read about the issues from the editors (Claire Goodess, Hans von Storch) who resigned because of a breakdown of the peer review process at that journal, that came to light with the particularly egregious (and well-publicised) paper by Soon and Baliunas (2003). The publisher’s assessment is here.
Update: Pulling out some of the common points being raised in the comments.
- HARRY_read_me.txt. This is a 4 year-long work log of Ian (Harry) Harris who was working to upgrade the documentation, metadata and databases associated with the legacy CRU TS 2.1 product, which is not the same as the HadCRUT data (see Mitchell and Jones, 2003 for details). The CSU TS 3.0 is available now (via ClimateExplorer for instance), and so presumably the database problems got fixed. Anyone who has ever worked on constructing a database from dozens of individual, sometimes contradictory and inconsistently formatted datasets will share his evident frustration with how tedious that can be.
- “Redefine the peer-reviewed literature!” . Nobody actually gets to do that, and both papers discussed in that comment – McKitrick and Michaels (2004) and Kalnay and Cai (2003) were both cited and discussed in Chapter
2 of3 the IPCC AR4 report. As an aside, neither has stood the test of time. - “Declines” in the MXD record. This decline was
hiddenwritten up in Nature in 1998 where the authors suggested not using the post 1960 data. Their actual programs (in IDL script), unsurprisingly warn against using post 1960 data. Added: Note that the ‘hide the decline’ comment was made in 1999 – 10 years ago, and has no connection whatsoever to more recent instrumental records. - CRU data accessibility. From the date of the first FOI request to CRU (in 2007), it has been made abundantly clear that the main impediment to releasing the whole CRU archive is the small % of it that was given to CRU on the understanding it wouldn’t be passed on to third parties. Those restrictions are in place because of the originating organisations (the various National Met. Services) around the world and are not CRU’s to break. As of Nov 13, the response to the umpteenth FOI request for the same data met with exactly the same response. This is an unfortunate situation, and pressure should be brought to bear on the National Met Services to release CRU from that obligation. It is not however the fault of CRU. The vast majority of the data in the HadCRU records is publicly available from GHCN (v2.mean.Z).
- Suggestions that FOI-related material be deleted … are ill-advised even if not carried out. What is and is not responsive and deliverable to an FOI request is however a subject that it is very appropriate to discuss.
- Fudge factors (update) IDL code in the some of the attached files calculates and applies an artificial ‘fudge factor’ to the MXD proxies to artificially eliminate the ‘divergence pattern’. This was done for a set of experiments reported in this submitted 2004 draft by Osborn and colleagues but which was never published. Section 4.3 explains the rationale very clearly which was to test the sensitivity of the calibration of the MXD proxies should the divergence end up being anthropogenic. It has nothing to do with any temperature record, has not been used in any published reconstruction and is not the source of any hockey stick blade anywhere.
Further update: This comment from Halldór Björnsson of the Icelandic Met. Service goes right to the heart of the accessibility issue:
Re: CRU data accessibility.
National Meteorological Services (NMSs) have different rules on data exchange. The World Meteorological Organization (WMO) organizes the exchange of “basic data”, i.e. data that are needed for weather forecasts. For details on these see WMO resolution number 40 (see bit.ly/8jOjX1).
This document acknowledges that WMO member states can place restrictions on the dissemination of data to third parties “for reasons such as national laws or costs of production”. These restrictions are only supposed to apply to commercial use, the research and education community is supposed to have free access to all the data.
Now, for researchers this sounds open and fine. In practice it hasn’t proved to be so.
Most NMSs also can distribute all sorts of data that are classified as “additional data and products”. Restrictions can be placed on these. These special data and products (which can range from regular weather data from a specific station to maps of rain intensity based on satellite and radar data). Many nations do place restrictions on such data (see link for additional data on above WMO-40 webpage for details).
The reasons for restricting access is often commercial, NMSs are often required by law to have substantial income from commercial sources, in other cases it can be for national security reasons, but in many cases (in my experience) the reasons simply seem to be “because we can”.
What has this got to do with CRU? The data that CRU needs for their data base comes from entities that restrict access to much of their data. And even better, since the UK has submitted an exception for additional data, some nations that otherwise would provide data without question will not provide data to the UK. I know this from experience, since my nation (Iceland) did send in such conditions and for years I had problem getting certain data from the US.
The ideal, that all data should be free and open is unfortunately not adhered to by a large portion of the meteorological community. Probably only a small portion of the CRU data is “locked” but the end effect is that all their data becomes closed. It is not their fault, and I am sure that they dislike them as much as any other researcher who has tried to get access to all data from stations in region X in country Y.
These restrictions end up by wasting resources and hurting everyone. The research community (CRU included) and the public are the victims. If you don’t like it, write to you NMSs and urge them to open all their data.
I can update (further) this if there is demand. Please let me know in the comments, which, as always, should be substantive, non-insulting and on topic.
Comments continue here.
1,074 Responses to “The CRU hack: Context”
-
1
-
2Leonard Herchen says:23 Nov 2009 at 12:45 AM
Gavin:
i’m going to repost the question in response to your question about my interest in a 10 year old graph.Thanks for responding. There are several reasons I care about the 10 year old graph, but the foremost one is simply credibility. I’m reading charge and counter charge in these websites, and my degree is in physics, not statistics or climate. So unless I can duplicate the work that you or MM do, I have to depend somewhat on my interpretation of credibility to decide who to beleive. Just like, if I was sick, I would go to the credible doctor, instead of getting my own medical degree. . The CA website threads ongoingly assert that climate science is closed system, doesn’t correct its errors and shouts down criticism. I don’t know if it is true or just internet breeze that I should ignore. So, I thought I would simply ask you the direct question and get the answer from you directly.
Even without a degree in climate science, I can see that mixing proxy data onto temperature data yields results that may not mean much, and the CA guys are making a big deal about this. So I wanted to check if it really is that simple. That is, was it done and not clearly acknowledged? And if so, why?
Again, I appreciate very much your attention to my comment.
[Response: Ok, last word on this before I turn in. This 10 year old graph is irrelevant to any current readings of the science. The caption describing what was done is unclear and should have been more complete. I have no personal knowledge of how smoothing was done in any of a hundred different variations of this particular theme. The rule should be that what ever is done, and for what ever reason, the description should match. The latest version of this kind of figure in IPCC AR4 is very clear about what is done, and it does not merge the two kinds of data. However, if you have two kinds of data showing similar things I am not surprised that people want to plot them together and I don’t see why that is – in principle – problematic. I’d be much more interested if this actually mattered. – gavin]
-
3J says:23 Nov 2009 at 12:46 AM
[Response: That would only be true if they had in fact produced convincing evidence that solar effects were larger than thought. They did not, and so the conclusion does not follow. – gavin]
I remind you of what is was replying to:
[Response: …Hint: if you assume that only one factor matters, then it is rather predictable that this is what you think is important.]
You seem to have gone backwards in your argument here.And I’ve already posted the link to the rebuttal of Benestad and Schmidt’s criticism.
[Response: (I’m Schmidt if you didn’t realise). You aren’t getting this. Scafetta’s work assumes that only solar can be responsible for the trends and very unsurprisingly comes up with a conclusion that it does! However, this doesn’t mean anything. In a proper attribution study you need to include all of the factors otherwise you risk conflating two elements which might both have a trend. In model results where you know the answer, Scafetta’s kind of technique does not work in extracting a true solar signal. Thus it demonstrates nothing about the real world. As for the ‘rebuttal’ we were promised a comment on the paper would be submitted within days of that blog posting. No such comment has emerged. Neither has a copy of Scafetta’s code that would allow us to reconcile our calculation and his. Maybe you’d like to complain that he’s ‘hiding’ something? – gavin]
-
4DavidCOG says:23 Nov 2009 at 12:58 AM
Could I recommend greenfyre.wordpress.com/2009/11/21/cru-hack-time-to-hit-back-hard/ to anyone who invests time arguing against the Deniers? We’re going to suffer them quoting words and phrases from these emails for quite some time, so it’s good to be clear on how to respond. Basically, just reply:
“So? How does that refute any of the *science*?”
-
5ccpo says:23 Nov 2009 at 1:08 AM
From the previous thread, about any credible anti-ACC science, and illogical behavior wrt ACC science:
re: ccpo 1031
Doesn’t the IPCC say there is a 90% chance that the change in temps is anthropogenic? Therefore doesn’t this mean there is a 10% chance it isn’t. Is that 10% chance of it not being anthropogenic “to any degree at all” for you?…
Regards
Michael
Comment by Michael — 22 November 2009 @ 9:56 PM
Actually, Mike, it does, but shouldn’t. There was a lot of discussion at the time about the political aspect of the final edition. The scientists wanted to use the term they defined to mean, essentially, zero doubt. The US gov’t, among others, objected and the next lower one was used.
I’d go get the sources, but what’s the point? 90% certainty for you people means, “Stop! Do nothing!!!” while for the rational it means, “90%? OH MY FREAKIN’ DEITY!”
An analogy (or two): 90% chance a car will hit you if you don’t move out of the road: you move.
90% it will rain: you take an umbrella.
90% you will die if you eat a poorly-cleaned puffer fish: you don’t eat it.
90% chance you will be in an accident and go to the hospital: you change your underwear.
A, what? 10%, or much less, chance you’ll be in a car accident in the next 5 years: you keep insurance.
A, what? 1% chance your house will burn down in the next 5 years: you keep insurance.
But, for you folk, a 90% chance of the increases in CO2/CO2e being human-caused, and dangerous to our current civilization and global sustainability? Party on, Dude!! That that chance is actually 100%, but scientists have been politically prevented from saying so in a documented form makes it that much more absurd.
If you can’t see your own ilogic staring you in the face, god help you. And us.
Re: Hank Roberts: “J, be skeptical..”
Thanks Hank, I think I am being. We now (if my last post was accepted) have the paper, it’s rebuttal and the response to the rebuttal.
I ask that you bear in mind the discussion this is part of. It’s was in response to a request for: “ANY scientific paper, or set of scientific papers, that disprove, to any degree at all, the anthropogenic influence on climate..”
Now, if you logically arrive at the requests assumption you have: “There is NO scientific paper, or set of scientific papers, that disprove, to any degree at all, the anthropogenic influence on climate..”
As you can see, this is far from your sensible request to be skeptical.
Comment by J — 22 November 2009 @ 11:42 PM
How is it not sensible? You deny ACC (Anthropogenically forced Climate Change) despite an amount of evidence that is truly astounding and offer two poor papers in response. Not only was my request sensible, it was prescient.
And you are right, I did assume you could not answer. This was sensible. There literally is not one single paper that disproves any aspect of what is currently known about ACC. Not one. At least, not that I know of or have ever heard about, and I’ve been asking for one for three years.
This simple fact should be the refrain from every ACC activist AND scientist: There is not even one scientifically sound paper that puts ACC in doubt, not even one small aspect of it.
What I find totally typical and unsurprising is that you don’t seem to understand what “disprove” means. You keep presenting papers you think challenges the science, but don’t, and don’t even come close to disproving anything.
I’m not surprised that scientists won’t make this same statement as they cannot possibly personally review every paper. But I can say this because it is true, and I’m not bound by absolute certainty. Very, very, very close is good enough. In fact, virtually all policy decisions other than climate change depend on a degree of certainty far, far below what we have for ACC, yet, we act on them all the time.
There is no logically tenable foundation for ACC denial. Caution? Sure. Denial? Truly a form of… well, it isn’t skepticism.
-
6Jeffrey Park says:23 Nov 2009 at 1:09 AM
Best Wishes as well to the RC moderators. The wave of disinformation is pretty intense. An ordinary reader of RC ought not be required to read all the purloined email to refute some of the trolls who express crocodile tears over “fraud” in climate science. When I learned that Hank Roberts had dug into one such claim and discovered that the “fraud” being discussed by the CRU scientists was one of the many flawed papers co-authored by the denialist Fred Singer, I could truly smell the crocodile.
-
7KTB says:23 Nov 2009 at 1:11 AM
Thanks for your efforts.
It’s good to have some responses, but at some point it’s probably best to let this be. The “skeptics” are far off in la-la land and are refusing to listen to any explanations for anything. They have their mind set as “the scientists are dirty and nothing can be trusted anymore”.One question came up which I haven’t seen answered or talked about.
There’s some controversy about tree ring data from new zealand oroko swamp. I don’t have the email about this but here is a graph and the “problem” is that it doesn’t show the hockey stick… The graph contains the emails number though (or the file contains the CRU data). So the presumption probably is that CRU has hidden this info somehowwep.fi/pic/OROKO_SWAMP_NEW_ZEALAND.jpg
[Response: No idea why this is interesting. I think this ended up as series 6 in figure 6.12 in AR4 “New Zealand tree rings”. And this has nothing to do with CRU in any case, it is from an Ed Cook paper. There is no discussion of any ‘problem’ with this data as far as I can see. – gavin]
-
8Dan E. Bloom says:23 Nov 2009 at 1:14 AM
This is why we are going to need polar cities in 500 years. Nobody ever listened to me. Pay attention now. pcillu101.blogspot.com
-
9Andrew says:23 Nov 2009 at 1:16 AM
As I understand, there are approximately 1000 emails in the leaked archive. Spanning over several years and comprising of communications among a number of people, 1000 emails tells me that this is a very small selection from the big picture.
People should keep that in mind, and make their own judgment considering the hacker’s intention of releasing these particular emails.
The trouble is, whatever context is added now, it still leaves a taint of suspicion about the rest of the unseen communication. Some will wonder if the kinds of communication we’ve seen are par for the course, the tip of the iceberg, or just cherry-picked for their eyebrow-raising value.
The bottom line should be whether any of the scientific conclusions are affected, and not what was meant by any particular word or exchange. Unfortunately, it’s too political for that.
-
10J says:23 Nov 2009 at 1:18 AM
>>>CCPO: “Seriously, even one, that in any way challenges anthropogenic warming as the cause of warming since 1850?”
Non-anthropogenic are not at least *part* of “the cause of warming since 1850?”Can you see this is an exaggerated claim?
It’s also cooled since 1850. And it’s been warming since the end of the last ice age. And it’s been warmer before 1850 – and colder.
Hyperbole detracts from credibility.
-
11Steve Bloom says:23 Nov 2009 at 1:23 AM
Gavin, you may want to edit #74 in the prior thread, noting the obscene pseudonym used by the commenter.
[Response: Maybe I’m getting old, but google gives no such interpretation as I can see….? – gavin]
-
12J says:23 Nov 2009 at 1:30 AM
[Response: (I’m Schmidt if you didn’t realise)…]
I do realize and I’m very aware that you are referring to your own work and that this is your site.
-
13Ian Wishart says:23 Nov 2009 at 1:43 AM
You guys just don’t stop spinning, do you? The context is there for anyone with English as a first language.
What part of stacking the peer review process don’t you understand is wrong?
What part of the email from Phil to Michael 1077829152.txt which reveals both men essentially in a position to peer review their own work (or challenges to their work or department’s) don’t you see as a conflict of interest?
[Response: Editors very commonly send papers critical of someones work to the author of the work being criticised. The editors also know that this needs to be weighed appropriately along with reviews from other parties. – gavin]
-
14BCC says:23 Nov 2009 at 1:48 AM
Re #74 :Gavin, say the poster’s name slowly. Hint: It starts “Hey, would ya…”, and yes, it’s lame.
Well, on the bright side, I was getting kind of tired of debunking the same old, same old. At least there’s something new to talk about…
[Response: Ah… – gavin]
-
15dukeofurl says:23 Nov 2009 at 1:51 AM
What was the ‘context of this comment
The other paper by MM is just garbage – as you knew. De Freitas again. Pielke is also losing all credibility as well by replying to the mad Finn as well – frequently as I see it. I can’t see either of these papers being in the next IPCC report. K and I will keep them out somehow – even if we have to redefine what the peer-review literature is !
[Response: The ‘mad Finn’ is very likely to be Timo Hameranta, a finnish lawyer, who for years regularly sent out hundreds of clippings of supposedly anti-GW abstracts to all and sundry. De Frietas was the editor on the Soon and Baliunas (2003) paper – but I’m not sure what is referred to here. MM is likely to be McIntyre and McKitrick (2005) in E&E (a very poor choice of journal if they wanted to be taken seriously). This last one was cited in IPCC AR4 (though against my suggestion in my review – based on it’s unclear status as a possibly un-peer-reviewed paper). – gavin]
[Further response: As noted below, the MM paper is McKitrick and Michaels (2004) and the first paper being discussed is Kalnay and Cai (2003). Both of which were cited and discussed in Chapter
23. – gavin] -
16Paul Middents says:23 Nov 2009 at 1:51 AM
Re #74
Just say the name quickly and you will agree with Steve Bloom.
You have the patience of Job.
Paul Middents
-
17Chris Byrne says:23 Nov 2009 at 1:54 AM
Gavin,
Have you had a look through the coding comments in Harry_Read_Me.txt? There’s a lot of speculation in the blogosphere about this. I have to admit I had a chuckle.
www.devilskitchen.me.uk/2009/11/data-horribilis-harryreadmetxt-file.html
[Response: That file is obviously just a notebook for someone piecing together work legacy code made by other people. Messy for sure, but certainly not the ‘final version’ of the code. It was probably produced in moving from the CRU TS 2.1 to 3.0 version (which is a completely separate data set from the standard HadCRUT numbers by the way) and involves a lot more interpolation. See here: www.cru.uea.ac.uk/cru/data/hrg.htm (when their server comes back up), also Mitchell and Jones (2005). – gavin]
-
18Mark says:23 Nov 2009 at 1:58 AM
This is crazzzzy. I don’t understand the logic of folks who are so quick to dismiss ACC. As an environmental activist it can be extremely frustrating to see the rules of inference perverted to such a degree. Think about this. If Albert Einstein created an equation that accounted for the relationship between mass and energy, or light and gravity, but failed to provide one that accounted for the expansion of the universe… would you deny the existence of the universe??? The denialistas are so full of non-sequiturs its scary…
-
19wayne davidson says:23 Nov 2009 at 2:17 AM
I think it better not to discuss hacked, stolen personal messages at all, even if they pertain to science. What use they have is dubious, and discussing them only encourages further hacking. Why not discuss public comments, and now commercials such as from friends of science, with their “10 year of cooling since 1998″ radio advert, due to the sun, Arghhh guys, we got work ahead not from stolen past messages which have meaning only to the writers, these were not meant for public discourse. We need discuss peer review papers and climate events and grasp their meanings, before enlisting the help of sordid hackers…
Read,:
www.huffingtonpost.com/2009/11/22/global-warmings-impacts-h_n_366994.html
That is a correct version of things, I personally confirmed this warming by independent refraction method.
Then in the same POST:
www.huffingtonpost.com/2009/11/20/inhofe-to-boxer-we-won-yo_n_365465.html
dont you think
Best wishes to RC and the relevant scientists – no professional deserves the violation of privacy done by these virtual thugs.
Please persevere; knowledge and information beats special interests in the long run.