The Sound Of All Hell Breaking Loose, Pt. 7

climategate2 120 Comments

The HARRY_READ_ME.txt file:

So, come with me on a wonderful journey as the CRU team realise that not only have they lost great chunks of data but also that their application suites and algorithms are total crap; join your humble Devil and Asimov as we dive into the HARRY_READ_ME.txt (thanks to The Englishman) file and follow the trials and tribulations of Ian “Harry” Harris as he tries to recreate the published data because he has nothing else to go on!
Thrill as he “glosses over” anomalies; let your heart sing as he gets some results to within 0.5 degrees; rejoice as Harry points out that everything is undocumented and that, generally speaking, he hasn’t got the first clue as to what’s going on with the data!
Chuckle as one of CRU’s own admits that much of the centre’s data and applications are undocumented, bug-ridden, riddled with holes, missing, uncatalogued and, in short, utterly worthless.
And wonder as you realise that this was v2.10 and that, after this utter fiasco, CRU used the synthetic data and wonky algorithms to produce v3.0!
You’ll laugh! You’ll cry! You won’t wonder why CRU never wanted to release the data! You will wonder why we are even contemplating restructuring the world economy and wasting trillions of dollars on the say-so of data this bad.

But don’t take their word. Here’s HARRY himself;

ARGH. Just went back to check on synthetic production. Apparently – I have no memory of this at all – we’re not doing observed rain days! It’s all synthetic from 1990 onwards. So I’m going to need conditionals in the update program to handle that. And separate gridding before 1989. And what TF happens to station counts?
OH FUCK THIS. It’s Sunday evening, I’ve worked all weekend, and just when I thought it was done I’m hitting yet another problem that’s based on the hopeless state of our databases. There is no uniform data integrity, it’s just a catalogue of issues that continues to grow as they’re found.

Upon this rests the wealth of nations.

120 Replies to “The Sound Of All Hell Breaking Loose, Pt. 7”

  1. ET: “…but the ‘Other World’ ignores it.”
    Not sure about that. The truly stupid are carrying on like nothing happened. But I have every faith in a politician’s sense of self-preservation … they are testing the wind.

  3. Indiana Homez writes:
    If a program isn’t documented line-by-line then it’s garbage, even if it’s working.
    That degree of commenting is a tad excessive although I agree that if one is writing a program for a paying customer then they have the right to know exactly what the source code does, especially if some neat hacks have been used in the code. BTW I always document my neat hacks as they seem far less obvious 10 years after I’ve written the program.
    When I was doing research my main contact with people for assistance was with electrical engineers as there was a lot of hardware design involved also. At the time the engineers had complete disdain for computer science students as they were obsessed with producing pretty code and documenting the obvious. An engineer or a results oriented programmer could produce a working program 10-100x faster than a computer science graduate.
    I think the key is knowing if the program is going to be used only once (to convert data from one tape format to another when the data acquisition program changes) or will be used multiple times and potentially extended. For one time use programs there is no reason to document. For a program that will be used by other people or extended then clarity is a virtue.
    As someone who has no problem reading code and figuring out what it does, I find comments often very irritating and as annoying as text which has two languages intermixed (like many Canadian bilingual documents). What I remembered after my last post was that I had paper documentation for all of my code as I hadn’t gotten to the point where I am now of just entering code directly into an editor to implement an algorithm. 30 years ago I would have a final flowchart of the program I was going to write and would translate the flowchart into a program or digital logic depending on which was the most efficient way to perform a given algorithm. I also had at least a few pages written showing what the program did and I never typed it in as it was primarily pictorial with a few text annotations. Interestingly, my handwriting back then was far more legible than now.
    One thing that strikes me about the CRU code is that it is NOT easy to read and the impenetrability of the code was what got me reminiscing about my early programming days. (I get the same feeling when I look at my 30 year old FORTRAN code). At least 30 years ago I had the excuse that on a 2.5 Mb disk drive there was little room for extraneous comments (and disk packs cost a couple of hundred dollars each) and the convolutions necessary to fit the code into the tiny PDP11 address space seem quaint now.
    CRU programmers can’t use these excuses as they are using Linux boxes and Beowulf clusters. The machines they program on probably have a few Gb of RAM and disk storage measured in hundreds of Gb. There is absolutely no penalty for adding comments to code now; if one doesn’t fee like typing one can always dictate the comments and store the result as a WAV file in the same directory as the program.
    Too bad you don’t like programming Indiana as I enjoy it as much now as when I first started 40+ years ago. If it paid as much as medicine I’d go back to doing it full time in an instant. Now it’s just a hobby that I indulge in when I have the time.

  5. Glenn Beck is all over this this afternoon. Even reading off an email that named himself…. heh heh!
    From: Michael Mann
    To: Andrew Revkin
    Subject: Re: mcintyre’s latest….
    Date: Tue, 29 Sep 2009 17:11:03 -0400
    All they have to do is put it up on their blog, and the contrarian noise machine
    kicks into gear, pretty soon Druge, Rush Limbaugh, Glenn Beck and their ilk (in this case,
    The Telegraph were already on it this morning) are parroting the claims.

  8. “That degree of commenting is a tad excessive although”
    Touché
    Fair enough, that being said, someone can be disingenuous and have a program output almost anything. A program that outputs a desired result through a rats nest of who knows what is worthless IMO. I suppose it’s akin to the correct answer for a math problem without documentation. Although the answer may be correct, without the work you can’t verify if it’s correct on merit, by fluke or by shenanigans.
    Anyways, a few macros here and there is the extent of my programming prowess. I didn’t own a PC until my first year of school so I struggled mightily in classes that others thought were a waste of time ie… hardware, programming, networking ect. Seriously, I had to be shown “right click, left click” I was so green, but because I was a strong math/physics student I was able to cope. Today I say “take a number” when I’m harassed for my tech skills. Perhaps under different circumstances I might have enjoyed that aspect of my education.
    “If it paid as much as medicine I’d go back to doing it full time in an instant.”
    But I thought doctors were in the biz to help people. You must be one of those “capitalist doctors” Owebumma loathes. I’ll be sure to check that my tonsils are still intact after the next time I see you. (kidding)

  9. Mann on Mann: uber-left GuardianUK.
    Mann: “”I hope it boomerangs back on the criminals.””
    …-
    “Michael Mann, director of the earth system science centre at the University of Pennsylvania, and a long-term target of sceptics, agreed the timing was suspicious.
    “What appears to have happened is that going into this monumental climate summit in a couple of weeks the other side, which does not favour taking action to combat climate change, resorted to an illegal smear campaign,” he said.
    “They are going through them and cherry-picking them for any word they can find that is cited out of context and can appear incriminating. I think it’s despicable.”
    He told the Guardian the emails – though embarrassing – did not undermine the body of science. “This doesn’t make any difference at all in degree of consensus on climate change,” Mann said. “I hope it boomerangs back on the criminals.””
    “Climate change email hacking to be looked into by University of East Anglia
    • Online publication seized on by denial bloggers
    • No evidence that data was falsified, says Met Office”
    http://www.guardian.co.uk/environment/2009/nov/23/climate-change-emails-uea

  11. Just for comparison, a business compiling information to conform to their environmental permit would never get away with such sloppy work. Just imagine the scandal if an oilsands co. refused to report their enviro data, fudged the results, had zero document control and then claim they lost their initial readings. The media, political outrage and bad PR would be the least of their problems – there would be fines and an immediate enviro audit.

  12. Thanks for all of the info Kate, I posted your most recent with a link back at the Flop and Wail earlier, it was deleted right away, then I posted your most recent again without the link and it was still there and with a post from “katewerk” right above, wonders never cease to amaze.
    Cheers

  13. “*BBC sat on Hacked CRU climate emails for a month!!!”
    …-
    “‘Climategate’ – CRU hacked into and its implications
    Paul Hudson | 13:07 UK time, Monday, 23 November 2009
    Very busy with forecast duties right now, but I do intend to write a blog regarding the UK Climate research centre (CRU) being hacked into, and the possible implications of this very serious affair.
    I will add comment on this page as soon as I can free up some time. But I will in the meantime answer the question regarding the chain of e-mails which you have been commenting about on my blog, which can be seen here, and whether they are genuine or part of an elaborate hoax.
    I was forwarded the chain of e-mails on the 12th October, which are comments from some of the worlds leading climate scientists written as a direct result of my article ‘whatever happened to global warming’. The e-mails released on the internet as a result of CRU being hacked into are identical to the ones I was forwarded and read at the time and so, as far as l can see, they are authentic.
    More later.”
    http://www.bbc.co.uk/blogs/paulhudson/2009/11/climategate-cru-hacked-into-an.shtml
    “*BBC sat …”
    http://www.freerepublic.com/focus/f-news/2392924/posts

  15. “He told the Guardian the emails – though embarrassing – did not undermine the body of science. “This doesn’t make any difference at all in degree of consensus on climate change,” Mann said. “I hope it boomerangs back on the criminals.””
    well he can’t forecast the climate but let’s all hope he clearly see his future.

  16. Fred:
    Interesting that Mann calls those who exposed this information criminals (hackers).
    Wonder what the British taxpayers, who have funded this criminal activity to the tune of 13.5 billion pounds, would think of the resistance to Freedom of Information as to where their dollars are going?

  17. “Wonder what the British taxpayers, who have funded this criminal activity to the tune of 13.5 billion pounds, would think”
    Set you Free . . .Once they snap out of their brain washed trance, they’ll be thinking that’s a whole lot of pints at their local they can’t afford and will be truly “upset”
    Maybe the CliamteGate hooligans will take up where the Football hooligans left off.

  20. I am reading the ommetns of Asimov here:
    http://www.tickerforum.org/cgi-ticker/akcs-www?post=118625&page=13
    I spent Saturday evening (OK I don’t have a life) casting an eye over harry_read_me.txt and came to very much the same view as Asimov.
    If you look at the date, it may be that the TIP-TOP Climate Scientists delegated the task of reconstructing the original data (which they realized they had lost:) to a student – who rightly got V.PO’d!! Perhaps he is the secret agent man.

  21. Loki – been at it for a while, eh? You kinda forgot that punch cards were the main reason for short variable names and limited comments. IIRC, F77 had a 6 character limit on variable/subroutine/etc names … I never had to get close to the limit.

  22. NOW … I’m willing to entertain the idea that the “Science is Settled” and the “Debate is Over”…..
    Waiting fore Suzuki to pop his head up.

  23. Anyone seen Suzuki since this broke? Has his melon-head exploded yet? And do we taxpayers get to recoup all the money cbc has paid this moron for all his enviro-crap for how many years??

  24. Suzuki and Gore were supposed to appear together for a forum at CBC’s Q Studio on Wednesday, has it been canceled?

  25. Goreacle Report: caw…caw…caw…buzzcawcaw >>>
    (Couldn’t resist the caws….)
    Q: “*”Will Al Gore Be Eating Crow?”
    Wall Street Journal: The two Canadians: Mc and Mc.
    Bravo to Stephen McIntyre and Ross McKitrick.
    …-
    “Global Warming With the Lid Off
    The emails that reveal an effort to hide the truth about climate science.
    ‘The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the U.K., I think I’ll delete the file rather than send to anyone. . . . We also have a data protection act, which I will hide behind.”
    So apparently wrote Phil Jones, director of the University of East Anglia’s Climate Research Unit (CRU) and one of the world’s leading climate scientists, in a 2005 email to “Mike.” Judging by the email thread, this refers to Michael Mann, director of the Pennsylvania State University’s Earth System Science Center. We found this nugget among the more than 3,000 emails and documents released last week after CRU’s servers were hacked and messages among some of the world’s most influential climatologists were published on the Internet.
    The “two MMs” are almost certainly Stephen McIntyre and Ross McKitrick, two Canadians who have devoted years to seeking the raw data and codes used in climate graphs and models, then fact-checking the published conclusions—a painstaking task that strikes us as a public and scientific service. Mr. Jones did not return requests for comment and the university said it could not confirm that all the emails were authentic, though it acknowledged its servers were hacked.
    Yet even a partial review of the emails is highly illuminating. In them, scientists appear to urge each other to present a “unified” view on the theory of man-made climate change while discussing the importance of the “common cause”; to advise each other on how to smooth over data so as not to compromise the favored hypothesis; to discuss ways to keep opposing views out of leading journals; and to give tips on how to “hide the decline” of temperature in certain inconvenient data.”
    http://online.wsj.com/article/SB10001424052748704888404574547730924988354.html?mod=googlenews_wsj
    …-
    “*Will Al Gore Be Eating Crow?
    When big money makers are involved in policies that can change the way every individual on the earth thinks, there will always be a few sceptics working on disproving the facts and figures that drive these written policies. We all know that Global Warming and carbon tax formulas shaped by the fact that everyone on this planet is responsible for the carbon gases emitted into our atmosphere is being construed by our world leaders today, but in light of the recent news, this might all change!
    According to many news sources around the world, someone hacked in to the files of the Climatic Research Unit based at the University of East Anglia. Apparently 61 megabytes of information was downloaded and posted onto the World Wide Web. Phil Jones Director of this research facility has acknowledged the fact that files are theirs. These files contain papers, documents, letters, and emails that outline information about manipulation of climate science data and the Intergovernmental Panel on Climate Change’s involvement. The (CRU) Climate Research Unit and Al Gore have close ties and the information based out of this institution plays a big role in Al Gore’s determination to tax carbon and collect money from everyone on this planet that emits carbon gases.
    The next question is, will the data and results from 10 years of scientific “number fudging”, manipulation, and corruption be further scrutinized or will all this be swept under the big red bureaucratic rug? In light of all this, I wonder how they will treat the hackers that exposed these hidden files to the rest world.”
    http://www.thesudburystar.com/Community/NewsDisplay.aspx?c=33660

  26. But what will Rumpy Pumpy Dumpty, EMPEROR of the EU PEOPLE, do in Copenhagen? Do you think his advisors have advised him that “Global Warming” is a crock? Will he demand immediate execution of the Hadley Crew … and a few non-Europeans to boot?
    …keeping fingers crossed :^)

  27. Its a glorious day in the neighborhood!!!!
    Leprechauns with Unicorns are disappearing everywhere.
    Nary a guitar in sight or Koo-Aide in place to drink.
    Could it be the reign of the Climate Space Reptiles is over?
    I love watching false prophets of pseudo-science. With even bigger lying “Teachers” being exposed. Top bad so much harm has been done to Nations plus the economy. Particularly to real environmental problems that we face . All in disrepute now.
    Such is the usual end of phantoms in the night.
    Y2K anyone? Any bets on 2012?
    JMO

  30. idd,
    “what our puplicly funded-Liberal run newsmedia CBC will not do”
    and the CTV News today?
    I watched for about a half hour, did I miss something? Probably the biggest News story…ever, that’s all.

  31. loki, indiana, et al:
    I remember doing FORTRAN on punch cards. Hell, I remember doing LISP on punch cards. ((((((That’s hard)))))) especially when the ribbons on the keypunch machines are worn thinner than Bambam’s excuses. I didn’t spend a whole lot of time on documentation then.
    But now, working in a corporate setting, where I know requirements are going to change frequently, I document everything, especially neat little “tricks” that help me accomplish things faster and with less code, but will GOTO hell in a handbasket if the data structure changes.
    Even SQL, which is pretty much self documenting, still needs a discussion of the database structure. It would just be like those dBA’s to change the file formats, add a new field, etc., and NOT TELL ANYONE ABOUT IT! Not that that’s ever happened to me, though…

  32. I haven’t a freakin’ clue what any of you are talking about; however, I think I like (a lot) whatever it is I think you’re talking about.

  34. Wafergate = instant Canadian MSM coverage. Long term effect: ZERO.
    Climategate = willful ignorance by Canadian MSM. Long term effect: the hollowing out of the Canadian economy.
    I don’t know, I guess the less we know the better for the MSM.

  36. Reuters comes through with the best lede ever.
    “Hacked climate e-mails awkward, not game change”
    Tooooooo fuuuuuuny.

  37. Line-by line comments are seldom needed, but if the code is insanely complicated, by all means put ’em in. But at a minimum, each function and routine should have a standard header showing
    Purpose and method
    variables used
    branch points, if any
    author, date, etc, plus explanations and names for any mods.

  38. Exactly blanks, at November 23, 2009 8:13 PM
    Cannot personally attest that CTV has not covered it on TV today, I don’t watch them at all anymore.
    CBC is what I monitor or check in on, as that’s what our tax dollars are paying for – unbiased relevant news. They are suppose to be the unbiased news source for all Canadians, but obviously they are not.
    FIRE.THEM.ALL.
    It’s not right that we have to pay for the voice of the LIbEral party of Canada, bad enough we’re forced to support the bloq party of Quebec, financially.

  39. I would love to see Jim Prentice bait the opposition with these leaked emails. Predictably they’ll howl and the MSM will jump all over it. It would be my hope that in all the MSM coverage the public will start to wonder if they’ve been had by politicians, the MSM, and those useful idiots at the CRU.

  41. I have to think that The Conservatives, the Sun and National Post are all “slow playing” this. Instead of diving in and making a big stink sometimes it’s best to drag it out for a long time. The Liberals are waiting for the big hit because they were 100% behind the global warming scam. I’d rather see them die of a thousand cuts than a stake to the heart.
    Get your ducks in a row and make them squirm. It will do far more damage than 1 big newspaper headline.

  42. “Suzuki and Gore were supposed to appear together for a forum at CBC’s Q Studio on Wednesday, has it been canceled?”
    When something gets this much attention online and then is TOTALLY ignored by the CBC, there is a problem. It leads me to conclude that the CBC is more interested in manipulating the news than reporting fairly.(Nothing new, I guess — I am just a bit surprised at how blatant this is.) I am less concerned about other outlets whom I don’t fund with tax dollars.
    Maybe they are embarassed because of the scheduled program on Wednesday — no matter — even a small, obscure story would help them appear more balanced. Too late for them to save face, I would say — unless they are willing to tackle this directly in the questions to Gore and Suzuki.

  43. Bruce,
    David Suzuki and Al Gore are still both listed on CBC’s Q schedule for Nov 25th.
    What’s even better is I find this gem:
    http://www.cbc.ca/q/blog/

    Q Blog
    November 18, 2009
    Ask Al Gore and David Suzuki…


    As you may have heard, on Wednesday (Nov. 25) Q has the privilege hosting two of the most prominent environmental activists and spokespeople in the world: former U.S. vice-president Al Gore and scientist and broadcaster Dr. David Suzuki. They will be here together in Studio Q.


    And we’d like you to get in on this unique conversation.


    If you have a question for Al Gore and David Suzuki, we’d love to hear it. If you had a chance to address these two, what’s the single most pressing question that you would you ask them?

  44. David Suzuki and Al Gore are still both listed on CBC’s Q schedule for Nov 25th.
    Is a lynch mob asking too much? I’d settle for tar and feathers then rode out of town on a rail.
    Hanging is probably too good for them.

  48. Enter, stage left: O’lOner…..
    “*…return with us now to those thrilling days” …-
    BBC asks: Can Obama Save the Planet?
    Yup, that’s the title of a documentary on BBC2 on Wednesday. Here’s the puff:
    Justin Rowlatt reports on whether President Obama is on target to keep his climate-change promises.
    Travelling across the States, using public transport only, Justin encounters coal miners and car manufacturers; activists and politicians; a pig farmer and a film star. He goes ice-fishing in Michigan, finds a thriving wind industry in the oil state of Texas and, in Detroit, drives a test car of the future.
    The film reveals two very different sides of America – the people who agree with their president that the US should be leading the world on tackling global warming, and those who see Obama’s plans as an attack on their liberties. And it’s not surprising that, in this divided America, the president’s push to limit greenhouse gas emission has become stuck in the mud of Washington politics.
    Not having seen the film, I have no idea which of the “different sides” of America Rowlatt identifies with. I’ve never been able to work out what the BBC makes of Barack Obama, and I can only deplore the implication of the vulgar cartoon that the Biased BBC website is running today:” (tOOn)
    http://blogs.telegraph.co.uk/news/damianthompson/100017668/bbc-asks-can-obama-save-the-planet/
    (*H/Tonto)

  49. inB at November 23, 2009 8:13 PM
    Lisp on punchcards? Bloody hell, that’s clever. I had the bracket completions of EMACS, also written in Lisp if I recall.

  50. BTW Does anyone realize that the editor used in MS Visuaql C/Basic/Whatever is the Brief editor?? I guess they bought them out, for a tidy sum I’m sure.

Proudly powered by WordPress & Skyrocket Themes

MENU

Navigation