So, come with me on a wonderful journey as the CRU team realise that not only have they lost great chunks of data but also that their application suites and algorithms are total crap; join your humble Devil and Asimov as we dive into the HARRY_READ_ME.txt (thanks to The Englishman) file and follow the trials and tribulations of Ian “Harry” Harris as he tries to recreate the published data because he has nothing else to go on!
Thrill as he “glosses over” anomalies; let your heart sing as he gets some results to within 0.5 degrees; rejoice as Harry points out that everything is undocumented and that, generally speaking, he hasn’t got the first clue as to what’s going on with the data!
Chuckle as one of CRU’s own admits that much of the centre’s data and applications are undocumented, bug-ridden, riddled with holes, missing, uncatalogued and, in short, utterly worthless.
And wonder as you realise that this was v2.10 and that, after this utter fiasco, CRU used the synthetic data and wonky algorithms to produce v3.0!
You’ll laugh! You’ll cry! You won’t wonder why CRU never wanted to release the data! You will wonder why we are even contemplating restructuring the world economy and wasting trillions of dollars on the say-so of data this bad.
But don’t take their word. Here’s HARRY himself;
ARGH. Just went back to check on synthetic production. Apparently – I have no memory of this at all – we’re not doing observed rain days! It’s all synthetic from 1990 onwards. So I’m going to need conditionals in the update program to handle that. And separate gridding before 1989. And what TF happens to station counts?
OH FUCK THIS. It’s Sunday evening, I’ve worked all weekend, and just when I thought it was done I’m hitting yet another problem that’s based on the hopeless state of our databases. There is no uniform data integrity, it’s just a catalogue of issues that continues to grow as they’re found.
Upon this rests the wealth of nations.
maz2 at November 23, 2009 9:50 PM
Maz, I expect that the BBC looks down their nose at The One, as they do all Americans. This is their snobbery; this is why I left the UK for Canada. I found Liberakl snopbbnery here, but I digress. The BBC will never pay an outright compliment to an American, even if they want the cream; it will alsways be sideways. It’sd a British Snob thing; N.Americans don’t understand the inmferiority complex of a failed people.
http://www.guardian.co.uk/commentisfree/cif-green/2009/nov/23/global-warming-leaked-email-climate-scientists
Monbiot here.
George is changing his surname from Moonbat to Monbiot.
His satire is pathetic.
But, we should give him some room for these words:
“It’s no use pretending this isn’t a major blow. The emails extracted by a hacker from the climatic research unit at the University of East Anglia could scarcely be more damaging. I am now convinced that they are genuine, and I’m dismayed and deeply shaken by them.”
this CRU fudging is, how can I put this, PERVERSE.
5 years from now and more IT professionals, academics, bloggers, some journalists, climatologists, historians are going to be still talking about the CRU cockup.
I joined university of western ontario in 1971 as a mainframe computer operator and went on to manage a PC retail store and teach college.
this CRU leak represents ALL that can go ‘worng’ in data analysis but even *beyond* that. unlike the old days when a misplaced parenthesis could crash the computer, when system analysts agonized over data type conversion, THIS is WILLFUL CALCULATED AND DELIBERATE effort to massage the numbers.
there will be books written and heads WILL roll.
mygod what have they done. what have they done. fraud, deception, lies, subterfuge, on and on.
any word yet who the hacker is?
ldd, Senator Inhofe will have the power to subpoena witnesses from the United States. I think Prof. Micheal Mann will have a lot of splaining to do.
I can see the questions now: “Prof. Mann, did you delete any emails or data knowing that a freedom of information request will soon be presented to you?”
Maybe we could get a joint US-UK-Canada hearings with Steve McIntyre and Ross McKitrick as the Canadian witnesses.
Update:
Google search results: Foi2009.zip
Thursday: 18
Friday: 1080
Monday: 5610
If you have not got your copy yet (via bit torrent):
http://www.mininova.org/tor/3168330
Woops, Thursday result should be 19.
Curious and others…..
I know it’s been posted before but….
I think we should call him/her a whistle blower, not a hacker.
My 2 cents to an unbelievable story. Like really. If I made this up last week would you believe it could happen?
Wall Street Journal is reporting that U.S. legislators are going to investigate.
http://online.wsj.com/article/SB125902685372961609.html
“Congressional Republicans have started investigating climate scientists whose hacked emails suggest they tried to squelch dissenting views about global warming”.
So. Sweet.
RW @ 9″43, at least they are honest about what their aim is.
Is this the point where some idiot say’s the dog ate my computer.
Jeff K, yes thanks for posting that reminder. I mentioned this in one of the previous Hell-Breaking-Loose threads, but it bears repeating. The person responsible for the ‘leak’ is not a hacker, but a whistle-blower.
IMO.
I maintain very large FORTRAN programs as a small part of my employment responsibilities. I must say than there are much better technologies available for complex programs rather than FORTRAN which is a dead language.
I would argue that no new work should be implemented in FORTRAN.
With regards to AGW, it has nothing to do with science. It is pure politics and we are losing. I would suggest people take a much stronger approach in fighting it before things come to open warfare.
Trust no one and check everything!
My $0.02
FORTRAN
havent used that since the days of the finite element method.
“You will wonder why we are even contemplating restructuring the world economy and wasting trillions of dollars on the say-so of data this bad.”
Yeah, no f#*king kidding!!!!
Fellow.Canadian: thanks for the CBC link. Here’s the question I left and we’ll see if it makes the list.
I’d like to know if you had any inside knowledge of the huge fraud being perpetrated by the climate “scientists” which was recently unearthed through the dissemination of the CRU emails last weekend? Given that the earth’s climate has been cooling for 10-15 years despite steadily increasing CO2 levels are you still prepared to mislead the Canadian public about a trace gas that is essential for life on earth? Are you aware of the far greater dangers to life on earth of potential asteroid collisions and how the process of deindustrialization that the two of you are proposing would eliminate any possibility of humans preventing such a catastrophe?
Good to see that there’s a lot of ex-FORTRAN programmers out there. Lorenzo, I wouldn’t call FORTRAN a dead language yet. There’s a huge amount of FORTRAN code that is open source, especially mathematical routines, that are a lot easier to incorporate into a FORTRAN program than to try to call from another language. I know how to do it, I just want to minimize the number of steps involved to get a working program. What finally got me off FORTRAN was first Hypercard on the Mac which made for very easy creation of forms and then I settled on VB which is now a “dead” language thanks to M$ stupidity. M$ is forcing people like me to go to Linux because there is no way I’m going to do a complete rewrite of my large codebase in VB to VB.NET. The few programs that I tried this on were so frustrating in terms of the vast amount of code that needed changing that I decided if I was going to go to an interpreted language it would be to Java.
What I do find incomprehensible is that someone would still write code the way I used to write it 30 years ago. FORTRAN is far more structured than it used to be and there is no excuse for writing unstructured code that is not expected to execute in a hard realtime system and runs on machines of todays incredible power. The only thing that would make sense is that someone at CRU was very lazy and had the FORTRAN code from the early 80’s and couldn’t be bothered to hire someone to recode it in another language. C has been around almost as long as FORTRAN and is more portable. Given the size of the grants that the people at CRU had over the last decade surely they could have sprung for a programmer or two; of course this might mean missing a conference in Tahiti. This would have resulted in fixing some of the bugs that were pointed out in the document which was the start of this thread.
The other possibility is that they stuck with FORTRAN knowing that it’s only aging programmers like myself who have much experience with the language and usually we’re way too busy to go through their code line by line to look for problems. By leaving the code maintenance to a grad student who’s only knowledge of FORTRAN prior to dealing with the code was from a web page on the history of computing, they’re going to create problems with data getting lost. This gives them a convenient scapegoat should something like the release of the CRU emails happen.
And speaking of legacy languages, while poking around in the DLL’s of my current medical billing program I determined that large chunks of it were written in COBOL! I assumed that this language had died decades ago as the only thing I thought it was good for was string manipulation of which there is very little in scientific computing. Also, while reorganizing my storage room this weekend I ran across 2 boxes of programs on punched cards that I had saved for some 40 years now and a paper tape of the same era. Fortunately I’d transferred these programs to a reel of 8 track tape which was an archival medium which would be around forever:-;
I hung around computational scientists several years ago (conferences and such). They mostly use Fortran, which has now been updated to Fortran 2003, with some parallel capabilities being proposed for the next version. LinPack (linear equations) is a major library written in Fortran. http://en.wikipedia.org/wiki/Fortran
At the time I noticed that many comp sci people assume that only comp sci people do programming (and that therefore only those languages in use by comp sci people are popular). Not true, by a long shot.
And I learned that some computational scientists are clueless when it comes to algorithms and data structures, let alone anything to do with software engineering (version control, change management, requirements analysis, qa and testing, etc etc). And they innocently/disdainfully believe that comp sci people must be dunces to spend so many years studying, since (to them) all one has to do is pick up a language manual and start coding…
My guess is that the East Anglia CRU scientists and doctoral candidates believe that when programming all one needs to do is pick up a language manual and start coding. If so, then the shoddy code and dubious data would not be surprising.
And then of course there are the temptations of (grant) money, honours and power.
This* seems to fit the beginning of the end of George’s AGW-fixation: Gaia’s legacy.
O’Gaia O’George: there is a parallel.
“*This is his sole legacy: a massive post-traumatic stress disorder.”
…-
“Monbiot issues an unprecedented apology – calls for Jones resignation
From Andrew Bolt, my “mate” down under at the Herald Sun, comes this surprise. I’ll have to say, it is to George Monbiot’s credit to do this. I embrace his first statement, because it succinctly sums up the situation:
It’s no use pretending that this isn’t a major blow. The emails extracted by a hacker from the climatic research unit at the University of East Anglia could scarcely be more damaging(1). I am now convinced that they are genuine, and I’m dismayed and deeply shaken by them.
– George Monbiot on his personal blog”
http://wattsupwiththat.com/2009/11/23/monbiot-issues-an-unprecedented-apology/#comments
…-
*O’narcissist:
“The “small people”, the “rank and file”, the “loyal soldiers” of the narcissist – his flock, his nation, his employees – they pay the price. The disillusionment and disenchantment are agonizing. The process of reconstruction, of rising from the ashes, of overcoming the trauma of having been deceived, exploited and manipulated – is drawn-out. It is difficult to trust again, to have faith, to love, to be led, to collaborate. Feelings of shame and guilt engulf the erstwhile followers of the narcissist. This is his sole legacy: a massive post-traumatic stress disorder.”
http://www.globalpolitician.com/25109-barack-obama-elections
Kate,
Great line at the end. Really highlights whats on the table.
Terence Corcoran at the National Post has the story. In fact the National Post has one full page devoted to climategate with Corcoran’s story linked on page 1.
Here’s the link to Corcoran.
http://network.nationalpost.com/np/blogs/fullcomment/archive/2009/11/23/terence-corcoran-after-copenhagen-the-end-of-the-science.aspx
We should all be thankful that PMSH and Jim Prentice had the sense to not drink at the trough of koolaid the liberals, NDP & blockheads are swilling from.