East Anglia University used to proudly host the Climate Research Unit. Now, perhaps, they wish that the well-heeled yet skeptical alumni didn’t know that this particular academic institution existed on their campus. EAU-CRU claimed to have the most comprehensive and well-ordered archive of climate data in the world. According to Lorrie Goldstein of the Toronto Sun, they have an archive that holds a certain volume of climate data.
The CRU assigned a programmer to archive and organize this sprawling agglomeration of data. They took this step so that they could retroactively validate claims to excellence that their professorate had already made. This was not only a moral shortcoming of this university; it was also a tactical error.
One of the sad ironies of both engineering and mathematics is that figuring out and organizing what you already have can be just as tough as discovering something new. Problems of this ilk, such as data normalization, and systems identification can give very competent practitioners of the scientific black arts splitting migraine headaches.
This programmer labored in the sterile vineyards of the CRU data warehouse from 2006 through this year. His methodical notes and working papers were collected into the standard read_me.txt file that all reasonable programmers include in the deliverable for any large project. His 274 page opus, HARRY_READ_ME.txt reads like a Dean Kuntz novel.
The now infamous programmer’s journal chronicles the man’s descent into ethical Gehenna. The programmer first realizes that his assignment is, as academics put it, “non-trivial.” Harry’s frustration shows below.
“But what are all those monthly files? DON’T KNOW, UNDOCUMENTED. Wherever I look, there are data files, no info about what they are other than their names. And that’s useless …” (Page 17)
- “It’s botch after botch after botch.” (18)
“Am I the first person to attempt to get the CRU databases in working order?!!” (47)
He then makes the shocking realization that a lot of this data is either counterfactual or even worse, made up from scratch.
– “COBAR AIRPORT AWS (data from an Australian weather station) cannot start in 1962, it didn’t open until 1993!” (71)
Finally, having realized that the data populating this database is not representative, he was forced to make a choice. He could have blown the whistle and declare this database fraudulent. He also could have played along and continued receiving remuneration for his efforts. From the quotes below, he became another working girl in the house of academic ill repute that was the East Anglia University Climate Research Unit. Here are some examples of Harry just going along to get along.
“What the hell is supposed to happen here? Oh yeah — there is no ‘supposed,’ I can make it up. So I have : – )” (98)
- “You can’t imagine what this has cost me — to actually allow the operator to assign false WMO (World Meteorological Organization) codes!! But what else is there in such situations? Especially when dealing with a ‘Master’ database of dubious provenance …” (98)
- “So with a somewhat cynical shrug, I added the nuclear option — to match every WMO possible, and turn the rest into new stations … In other words what CRU usually do. It will allow bad databases to pass unnoticed, and good databases to become bad …” (98-9)
And so it goes in modern academia. Like the corrupt, self-serving police officers in the novel LA Confidential, people get forced to bend or break. The hard choices forced by the ethical squalor amongst the leadership of a major academic enterprise debased otherwise decent people into similar moral decline.
Harry had to choose between his own soul and the continued success of his team. Yet, even in the end, after he had sacrificed his integrity; reality still wouldn’t budge a single degree Fahrenheit. As he continued to try and bring order to dishonesty, the vast ineluctability of the project seemed to overwhelm and demoralize Harry.
– “OH F— THIS. It’s Sunday evening, I’ve worked all weekend, and just when I thought it was done, I’m hitting yet another problem that’s based on the hopeless state of our databases.” (241).
- “This whole project is SUCH A MESS …” (266)
All of that, for a graduate stipend, or maybe a PostDoc… Things like this are why I enraged most of the blogosphere by demanding that science be put under a regulatory regime similar to Sarbanes-Oxley.
Daniel Horowitz
Neil Stevens
Moe Lane
Steve Maley
Jake Walker
RMJ: Do you think Harry could be the leaker? nt
Steve Maley (Diary) Tuesday, December 1st at 5:36PM EST (link)The blogger formerly known as ‘Vladimir’.
Things That Me Go Hmmm...
Repair_Man_Jack (Diary) Tuesday, December 1st at 6:16PM EST (link)That could be the best nt post on RS this week.
” I side impenitently with the human race against the modern reformer.” – C.S. Lewis
great report RMJ - nt
Mike gamecock DeVine (Diary) Saturday, December 5th at 8:30AM EST (link)Mike DeVine’s Examiner.com, Charlotte Observer and The Minority Report columns
“One man with courage makes a majority.” – Andrew Jackson
Thanks for this diary Repair it is very much appreciated....
JadedByPolitics (Diary) Tuesday, December 1st at 6:30PM EST (link)I know I have KNOWN this was ALL a LIE but to actually see it just leaves me breathless to the extent of the FRAUD. It is almost like how I felt when Justice Ginsberg ADMITTED that Roe v Wade in her mind and leftists like her and of course the mother of the abortion Margaret Sanger that abortion was for the purpose of deleting certain segments of society.
You know its TRUE but to hear it or read it just shakes your very foundation. It just shows P.T. Barnum was correct there is indeed one born every minute and they now of course are in the billions around the world!
Unified Patriots – How-To:
Activists Taking Action
What leaves me breathless is the left's lies about why it doesn't matter.
nessa (Diary) Tuesday, December 1st at 8:01PM EST (link)WTF?! Media Matters claims they’ve already debunked the whole thing. Nothing happening here folks, just move along now.
The righteous indignation should be driving us rioting into the streets, hunting every global warming supporter known to man and applying a thick coat of tar and feathers. McCarthy type hearings should be happening in congress with algore excoriated and pilloried. They can finish by confiscating all his ill gotten gains and paying down the deficit. Then give him to us so we can tar and feather his a$$ too!
Once we’ve run the global warmers out of town on a rail we can shift fire to the few remaining enviro-wackos, and make sure we get every scientist who ever, ever studied a cow fart, I’ve got something special for those idiots!!
“If you love wealth more than liberty, the tranquility of servitude better than the animating contest of freedom, depart from us in peace. We ask not your counsel nor your arms. Crouch down and lick the hand that feeds you. May your chains rest lightly upon you and may posterity forget that you were our countrymen.”—Samuel Adams
Contributor to Unified Patriots
teh twitter
The left can claim what they want
jackbenimble (Diary) Wednesday, December 2nd at 10:49AM EST (link)The left can claim what they want but the fact remains that the hit counts at all of the sceptic web sites like Climate Audit, Watts Up With That, Junk Science and Climate Depot are going through the roof. And Drudge has been featuring these stories on his front page for a week. The word is getting out fast and the left’s silly denials are like trying to piss into a hurricane. Nobody is listening to them and when they do, they realize that they are part of the problem. Liars covering up for liars.
“I repudiate the idea of voting for a Democrat
"For what shall it profit a man if he shall gain the whole world,
civil truth (Diary) Tuesday, December 1st at 6:57PM EST (link)…and loses his own soul? Or what shall a man give in exchange for his soul?”
The road to perdition oft times is a series of steps, one leading to the next, each one small – that one doesn’t see how far one has strayed from the strait way.
But if you mercifully see where you’ve traveled, then the hard decision is whether to turn back or plunge headlessly ahead.
This is a revealing chronicle of that process.
The greatest evil…is conceived and ordered (moved, seconded, carried, and minuted) in clean, carpeted, warmed, and well-lighted offices, by quiet men with white collars and cut fingernails and smooth-shaven cheeks who do not need to raise their voice. Hence, naturally enough, my symbol for Hell is something like the bureaucracy of a police state or the offices of a thoroughly nasty business concern. -C.S. Lewis
http://www.gmsplace.com/
I May Not Have Hit The Scriptural Reference,
Repair_Man_Jack (Diary) Tuesday, December 1st at 7:07PM EST (link)But that was clearly where my mind traveled when I finished rereading all of the quotes in the original news article.
” I side impenitently with the human race against the modern reformer.” – C.S. Lewis
Very well said, civil_truth
6eorge Jetson (Diary) Monday, December 7th at 8:34AM EST (link)“You cannot distinguish between extreme incompetence and conspiracy”
-a variant of a quote attributed to Napoleon
The challenges of describing the data measurements in a manner that retains their meaning is definitely non-trivial. Much, much more difficult than the purchase of Terrabytes of disk space.
Such organization requires significant data management skills. Why did the CRU not bother to retain the original data when it moved? Perhaps it was an over-the-top attempt to cover brazen intentional fraud. Perhaps it was a poorly-considered recognition of the dubious state of the data. Or perhaps it was an excuse to hide from the embarassing state of the data.
How despite its bountiful funding, the CRU got to this state is certainly a cocktail of many individual stories, some of them based on incompetence, some of them based in malice/fraud.
Very well said, civil_truth
It's Dean KOONTZ.
barrypopik (Diary) Wednesday, December 2nd at 2:12AM EST (link)“His 274 page opus, HARRY_READ_ME.txt reads like a Dean Kuntz novel.”
Good work, Repair Man. nt
redneck_hippie (Diary) Wednesday, December 2nd at 8:52PM EST (link)Guess What!! NPR finally mentioned "The Stolen E-mails".
jayburd (Diary) Wednesday, December 2nd at 11:50PM EST (link)Then they promptly soft-spun it as “scientists with bad manners” and did not mention the code.
One of my heroes- Ralph Smeed’s blog- http://smeedonstate-ism.com/index.htm
“What’s the matter? Don’t you want to win the war?” – Capt. John Birch
“If the Nation can issue a dollar bond it can issue a dollar bill.
The element that makes the bond good makes the bill good also. The
difference between the bond and the bill is that the bond lets the
money broker collect twice the amount of the bond and an additional 20%.Whereas the currency, the honest sort provided by the Constitution pays nobody but those who contribute in some useful way. It is absurd to say our Country can issue bonds and cannot issue currency. Both are promises to pay, but one fattens the usurer and the other helps the People.” – Thomas A. Edison
I pointed out to NR
baserunr (Diary) Thursday, December 3rd at 7:04PM EST (link)(not NPR) a few days ago, the real story is in the programmer notes. The e-mails, as bad as they are, are not the bulk of the story, IMHO. The programmer notes make it explicitly clear that the data is useless, and thus all conclusions and machinations resulting from it are useless as well. Wouldn’t it be nice if the MSM devoted as much air time to this issue as they did to propagating the hoax in the first place?
But hurry, we have to pass Cap & Trade, before the climate damage is irreversible. What a load….
“The day you think you know it all is the day your trouble starts.”
This was my favorite find in the "data"
izoneguy (Diary) Friday, December 4th at 6:27PM EST (link)The point cannot be made often enough: Modern liberalism, as embodied in the Obama presidency, is the defender of the status quo. And the status quo is a road to economic ruin. Political forces cannot redistribute the wealth that the economic system does not produce.
Or this letter to Congressman Joe Barton from Michael Mann
izoneguy (Diary) Friday, December 4th at 6:37PM EST (link)Titled:
MannHouseReply.pdf
July 15, 2005
Joe Barton, Chairman
House Committee on Energy and Commerce
Ed Whitfield, Chairman
Subcommittee on Oversight and Investigations
2125 Rayburn House Office Building
Washington, D.C. 20515
Dear Chairman Barton and Chairman Whitfield:
This letter responds to your letter of June 23, 2005, which seeks information on issues relating to my research on the historical record of temperatures and climate change. Your letter lays out a number of “concerns” about the research my colleagues and I have conducted about global warming. Your letter also inquires about the role I
played in the preparation of the United Nation’s Intergovernmental Panel on Climate Change Third Assessment Report (the so-called “TAR”).
I will address each of your questions in turn. Before doing so, however, let me state that my research findings, which support the conclusion that the earth’s surface is warming, and that recent warming is due in large part to human influences, are consistent with the overwhelming scientific consensus on climate change.
(Blah, blah, blah….)
It also bears emphasis that my computer program is a private piece of intellectual property, as the National Science Foundation and its lawyers recognized. The National Science Foundation — the government agency that establishes policy in this area — has
confirmed that my colleagues and I have met every requirement of transparency and openness in our research. My research is all based on data sets regarding the Earth’s climate that are freely and widely available to all researchers.
(Even if it is flawed and made-up data!! LOL)
For the Committee’s convenience, I have sent along with this letter copies of key scientific articles referred to in this letter. Please let me know if you have questions.
Respectfully submitted,
Michael E. Mann, Ph.D.
Associate Professor and
Director of Earth System Science Center
Department of Meteorology
The Pennsylvania State University
TA DA – And YES here are the questions!!!!
http://joebarton.house.gov/Default.aspx
Barton presses Obama Adminstration to unearth possible links to ClimateGate
“We note that employees and researchers supported by the Department of Energy, the National Nuclear Security Administration and/or Lawrence Livermore National Laboratory figure prominently in the emails, especially LLNL scientist Dr. Benjamin Santer and DOE-funded scientist Dr. Phil Jones of the University of East Anglia, who appears to be at the center of the email collection,” Barton and Walden wrote to Chu. “Some of the emails raise concerns about researchers’ efforts to evade or circumvent the Freedom of Information Act, and DOE’s compliance with FOIA in responses to such requests. The discussions among researchers reflected in the emails also raise questions concerning the integrity of some of the DOE-funded research at issue in the emails and the researchers’ adherence to federal data sharing guidelines, policy, and oversight.”
The point cannot be made often enough: Modern liberalism, as embodied in the Obama presidency, is the defender of the status quo. And the status quo is a road to economic ruin. Political forces cannot redistribute the wealth that the economic system does not produce.
You Hoo - President Obama, maybe you need to cancel your trip
izoneguy (Diary) Saturday, December 5th at 10:13AM EST (link)As Sarah Palin has suggested….
Unless of course you need to get embarassed yet again on the world stage? I guess you could always take a bow?
Met Office to re-examine 160 years of climate data -
The Government is attempting to stop the Met Office from carrying out the re-examination, arguing that it would be seized upon by climate change sceptics. – HA HA HA HA HA – You think!!!!!
http://www.timesonline.co.uk/tol/news/environment/article6945445.ece
The point cannot be made often enough: Modern liberalism, as embodied in the Obama presidency, is the defender of the status quo. And the status quo is a road to economic ruin. Political forces cannot redistribute the wealth that the economic system does not produce.
Perhaps Zero's trip will draw even greater attention to the sCRU up
6eorge Jetson (Diary) Monday, December 7th at 8:03AM EST (link)Just like his “the police acted stupidly” in the Harvard professor Henry Gates affair (GatesGate ? Gate^2 ? )
Harry's experience reminds me of my own
mustango (Diary) Monday, December 7th at 1:41PM EST (link)At my first-ever full-time, “real world” job, I was assigned with the task of creating a statistical package for tracking manufacturing variations. It had all the bells and whistles and features whose use I could barely grasp, like something called kurtosis.
Anyway, in the end it was all nice and neat and worked beautifully, but then it came back with a complaint from the customer. “How come when we change this value, that other value changes?” I replied simply to the sales guys that forwarded the complaint, that that was how the math worked.
For some reason this did not satisfy someone, and in the end I was directed to rewrite the package in such a way as to permit inconsistent tolerance values to be entered.
I don’t know if that experience contributed to the illness I contracted soon after that cost me that job and set back my career by years, but I’m sure it didn’t help.
As far as I know the product was only used internally by corporate entities, and for all I know the bogus feature may never have been used in a meaningful way.
All I know is that it was an early lesson as to how scientific truth can get subjugated by those who think they know better than the numbers.
“I just miss — I miss being anonymous.” — Barack Obama
I fully sympathize, Mr. President. I miss you being anonymous too.