So, come with me on a wonderful journey as the CRU team realise that not only have they lost great chunks of data but also that their application suites and algorithms are total crap; join your humble Devil and Asimov as we dive into the HARRY_READ_ME.txt (thanks to The Englishman) file and follow the trials and tribulations of Ian “Harry” Harris as he tries to recreate the published data because he has nothing else to go on!
Thrill as he “glosses over” anomalies; let your heart sing as he gets some results to within 0.5 degrees; rejoice as Harry points out that everything is undocumented and that, generally speaking, he hasn’t got the first clue as to what’s going on with the data!
Chuckle as one of CRU’s own admits that much of the centre’s data and applications are undocumented, bug-ridden, riddled with holes, missing, uncatalogued and, in short, utterly worthless.
And wonder as you realise that this was v2.10 and that, after this utter fiasco, CRU used the synthetic data and wonky algorithms to produce v3.0!
You’ll laugh! You’ll cry! You won’t wonder why CRU never wanted to release the data! You will wonder why we are even contemplating restructuring the world economy and wasting trillions of dollars on the say-so of data this bad.
But don’t take their word. Here’s HARRY himself;
ARGH. Just went back to check on synthetic production. Apparently – I have no memory of this at all – we’re not doing observed rain days! It’s all synthetic from 1990 onwards. So I’m going to need conditionals in the update program to handle that. And separate gridding before 1989. And what TF happens to station counts?
OH FUCK THIS. It’s Sunday evening, I’ve worked all weekend, and just when I thought it was done I’m hitting yet another problem that’s based on the hopeless state of our databases. There is no uniform data integrity, it’s just a catalogue of issues that continues to grow as they’re found.
Upon this rests the wealth of nations.
The smartest men in the room …
Not surprisingly, this is exactly the kind of crap (with an additional layer or twelve of obfuscation) that CA has detailed over and over.
Either these jackasses are the stereotypical scatterbrained ivory-tower eggheads misplacing their data along with their spectacles every fifteen minutes, or their zeal to avoid embarrassing FOIA requests by deleting files got the better of them.
Bre-X redux
Oh dear.
I feel his pain, I deal with “dirty data” all the time in my job, it’s grim.
And heck, in my world people are trying to get at the truth because the truth has a $ prefix and if you get it wrong even slightly, you get fired, and maybe your company goes out of business.
What’s funny is that there wasn’t even an attempted to establish data set “v1.0” a nice clear, un-adulterated “let the chips fall where they may” data set that you can then brutalize in a systematic manner to get it to fit your assumptions and then release as v2.0 “the official corrected data”. What kind of incompetence is this?
They are dealing with crap base data, piling crap corrections upon crap corrections, both in sign and in time, making dubious assumptions,and ignoring massive margins of error. My professors would have kicked my ass out of both my advanced degrees had I done that and drawn a conclusion that was not: “This is crap, at the least, we need better data, or better yet a new thesis topic.”
It really makes me sad to think of the billions wasted on the research alone…
Like to give a big
“shout out”
to the hero that got this data onto the internet for public viewing, as it should have been all along.
Bravo, whoever you are.
And thanks Kate for doing what our puplicly funded-Liberal run newsmedia CBC will not do on this most important issue facing Canadians and the world.
Or maybe they’re busy looking for a way to blame Bush, again, somehow, someway, eh?
Such a solid foundation to build the Global Scientific Consensus on.
After all, we are talking about changing the entire planetary economy and the future credibility of Climate Scientists, in fact the credibility of all Science, is on the line.
I for one will rest easy now that the basis for the consensus is Rock Solid.
I can feel a good night’s sleep coming on.
I love the solution for arithmetic results that go beyond Fortran’s bounds: just replace them with a static acceptable value, call it “fixed,” and forge ahead!
———-
17. Inserted debug statements into anomdtb.f90, discovered that a sum-of-squared variable is becoming very, very negative!
. . .
forrtl: error (75): floating point exception
IOT trap (core dumped)
..so the data value is unbfeasibly large, but why does the sum-of-squares parameter OpTotSq go negative?!!
Probable answer: the high value is pushing beyond the single-precision default for Fortran reals?
. . .
Action: value replaced with -9999 and file renamed:
pre.0312031600H.dtb (to indicate I’ve fixed it)
.dts file also renamed for consistency.
anomdtb then runs fine!! Producing the usual txt files.
“Back to the gridding. I am seriously worried that our flagship gridded data product is produced by Delaunay triangulation – apparently linear as well.
As far as I can see, this renders the station counts totally meaningless.
It also means that we cannot say exactly how the gridded data is arrived at from a statistical perspective – since we’re using an off-the-shelf product that isn’t documented sufficiently to say that.
Why this wasn’t coded up in Fortran I don’t know – time pressures perhaps? Was too much effort expended on homogenisation, that there wasn’t enough time to write a gridding procedure? Of course, it’s too late for me to fix it too. Meh.0”
DEAD IN THE GLOBAL WARMING GATE INFERNO…
Welcome to data Hades ladies and gentlemen, global climate terror is upon us.
And so rests the fate of the planet, on those who can’t code FORTRAN because its “too late for me to fix it.”
Cheers
Hans-Christian Georg Rupprecht, Commander in Chief
1st Saint Nicolaas Army
Army Group “True North”
More programming beauty (my bolds)
————–
% Compiled module: RDBIN.
% Compiled module: STRIP.
yes
% Variable is undefined: NF.
% Execution halted at: RDBIN 68
. . .
So what is this mysterious variable ‘nf’ that isn’t being set? Well strangely,it’s in Mark N’s ‘rdbin.pro’. I say strangely because this is a generic prog that’s used all over the place! Nonetheless it does have what certainly looks like a bug:
. . .
In other words, ‘nf’ is set in the first conditional set of statements, but in
the alternative (starting on #62) it is only set AFTER it’s used (set #73,#74; used #68). So I shifted #73 and #74 to between #64 and #65, and..
with precompiling to pick up the local version of rdbin, too.. it worked!
Er, perhaps.
We’re still not hearing much from “the science is in” crowd.
C’mon guys, were you all irradiated by the “mushroom cloud” as it was described by a real scientist rather than a tarot card reader?
No three cheers for the Nutley CRU?
Speaking of the Sound Of Hell Breaking Loose.
Here’s another treasure trove of progressivism about to be acid dripped into the public domain.
Gawd this is a good time to be fighting the good fight . . .
http://tinyurl.com/yd9dnu3
Such capers are only confined to ‘climate scientists’? Dream on …
Their careers as Microsoft programmers are assured (my bolds):
————
Ulp!
I am seriously close to giving up, again. The history of this is so complex that I can’t get far enough into it before by head hurts and I have to stop. Each parameter has a tortuous history of manual and semi-automated interventions that I simply cannot just go back to early versions and run the update prog.
I could be throwing away all kinds of corrections – to lat/lons, to WMOs (yes!), and more.
So what the hell can I do about all these duplicate stations? Well, how about fixdupes.for? That would be perfect – except that I never finished it, I was diverted off to fight some other fire. Aarrgghhh.
I – need – a – database – cleaner.
What about the ones I used for the CRUTEM3 work with Phil Brohan? Can’t find the bugger!! Looked everywhere
. . .
I think the comment for *this* program sums it up nicely:
[the following are comments embedded in one of their Fortran programs – Matt]
‘ program postprocdupes2
c Further post-processing of the duplicates file – just to show how crap the
c program that produced it was! Well – not so much that but that once it was
c running, it took 2 days to finish so I couldn’t really reset it to improve
c things. Anyway, *this* version does the following useful stuff:
c (1) Removes and squirrels away all segments where dates don’t match;
c (2) Marks segments >5 where dates don’t match;
c (3) Groups segments from the same pair of stations;
c (4) Sorts based on total segment length for each station pair’
You see how messy it gets when you actually examine the problem?
Expecting the MSM to expose this is as rational it as expecting the Waffen SS and/or Gestopo to support the coup following the attempt to assassinate Adolph Hitler.
Yeah…..we were much smarter then…..
Thanks big time, Kate, for the absolutely superb coverage here at SDA in the last couple of days. SDA is going to have a lot more impact than, say, the Star. I’ve seen in the past how certain issues have been, erm, forcefully inserted at the end of Kate’s arm into various Canadian media – SDA does get results.
This Hadley story is another one of those watershed moments that prove the supremacy and, increasingly, influence of the blogosphere relative to the traditional media. Speaking of the media, Ed Driscoll has a great piece up – ““All The News That’s Fit to Bury” at Pajamas media. Excerpt:
“Seeing as they each impact key pillars of what today passes for liberalism, there seems to be more than a few connections between the recent ACORN stings by Giles, O’Keefe and Breitbart, and the recent hacking of the emails of the University of East Anglia Climate Research Unit, or ‘Global WarmingGate, as Charlie Martin dubs it elsewhere at Pajamas. Not the least is that they each sent the legacy media into full gatekeeper mode, hoping to prevent exciting, important news of current events from ever reaching their readers. Or perhaps, like the scandal last year involving John Edwards, sitting on the stories for so long, while making claims that they have to endlessly research them to verify their authenticity — Keep rockin’! — that when the legacy media decides to go ‘public’ with news that everyone already knows, they can dramatically dilute the ultimate impact of these stories.”
Driscoll links to Candace Moore at NewsBusters, who writes about the way the legacy media moves the proverbial goalposts in an utterly shameless, straight-faced way:
“On November 22, while responding to the growing scandal about alleged proof that global warming is a hoax, the Times brushed it off with a puzzling claim that science should have no bearing on climate legislation.” Moore notes that six weeks ago, the LA Times criticized George Bush for “hiding scientific data that could be used to sway the debate over the legislation…(…)…When scientific findings were there to warn that global warming would kill the planet, the Times was quick to support it; when science was later found to be riddled with tricks that tainted its credibility, climate legislation was suddenly all about fixing the economy.”
Rand Simberg, also at Pajamas media:
“When scientists become politicians but continue to pretend to be doing science, that is the real crime. The theory being promoted by these men was being used to justify government actions that would result in greatly diminished future economic growth of the most powerful economy on earth (and the rest of the world as well). It would make it more difficult and less affordable to address any real problems that might be caused in the future by a change in climate, whether due to human activity or other causes. It could impoverish millions in the future, with little actual change in adverse climate effects. And when such a theory has the potential to do so much unjustified harm, and it has a fraudulent basis, who are the real criminals against humanity?”
Verily.
Oh man. I cannot keep up. This mushroom cloud is soooo HUGH!!!
Oh man. I cannot keep up. This mushroom cloud is soooo HUGH!!!
Posted by: Louise at November 23, 2009 2:52 PM
I hear you, Louise. I’m savoring every minute of this. These fraud artists have everything that’s coming to them.
Only an elite academic would use so many words to state the obvious – DATA NFG.
I guess the NY Times must think it made a mistake with the Daniel Ellsberg leak of the Pentagon Papers. Sure!
Ideology trumps truth.
“This mushroom cloud is soooo HUGH!!!”
Boom de adda, Boom de adda,
Boom de adda, Boom de adda,
Boom de adda, Boom de adda,
I’m lovin it too.
Yup . . it is a good day. Too bad the CBC, the Star, the G&M, CTV haven’t even clued into the story yet so more people could enjoy it.
Maybe they are too embarrassed from drinking the Al Gore/David Suzuki Kool-Aid for so many years, of being the mouthpiece for every AGW nutter & whackjob progressive who hated our modern world.
Or maybe they are just frik’n incompetent.
Not a peep out of them. Our Peepless MSM.
Mount Pinatubo!! This time with very selective and only metaphorical kills.
You know its bad when even AGW Believers are turning on The Team.
http://tinyurl.com/y9ogxqx
“Pielke’s views are cogently expressed, coherent, backed up by numerous studies in the field–and to a considerable extent, they have been marginalized by critics from The Team, who viewed him as dangerous, to the extent that he could take the focus off their obsession, human emissions of CO2. It didn’t matter that Pielke believes (and has often written) that human emissions of CO2 are a powerful factor. It didn’t matter that he is as far from a skeptic as you can possibly be. It didn’t matter that his academic credentials and publishing record rivalled, if not surpassed, their own.
The Team viewed Pielke (as they did so many others) as an enemy. The released emails contain references to him as an albatross, and tried to make hims seem like one who had lost all credibility”
I hope part of what’s coming their way is jail time.
What Idd said.
1296!
ALL DISCUSSION of global warming, climate change, IPCC, tipping points, Copenhagen, CO2 emissions, cap & trade, ETS, etc., etc., ceases – NOW!
The UN (ugh!)is tasked to fund and set up 1296 super-duper surface temperature/CO2 reporting stations – on land and at sea. One station at the convergence of each 10 whole degrees Lat & Long. 36 X 36. Difficult, but doable.
All other, existing stations are shut down. Any station that falls into an existing Urban Heat Island is relocated (by consensus) to a nearby spot that qualifies for proper, uncontaminated recording, and is protected as such.
NASA or whomever is responsible for collecting atmospheric temperatures, funds and sets up the space equivalent recording points – above, below or in the “GHG” layer, wherever – but in consort with those 1296 points!
The ensuing 50-year record (pick a time frame) is transparently and universally broadcast from every state capitol in the world – daily, weekly, monthly, and reposited in a central data base.
Not one further red public cent, any time, anywhere, is channeled to anyone who may happen to have a theory or hypothesis with regard to AGW, CC, GHG, etc.
All existing pollution programs or regulations stay in effect, but are not enhanced or diminished. The world unfolds as it will for that time frame.
At the end of 50 (or whatever) years, the world will either be cooler, warmer or remain unchanged.
THEN we MAY or NOT want to do something about it.
The charlatan “warmers” will be unemployed; the “deniers” will go back to productive science; the rest of us can continue to smite each other about the head over religion, health care reform, voter fraud, multiculturalism, nuclear energy, immigration, racism, alternative energy, burkas or pandemics, i.e., sane topics of discourse!
GIGO
I’m once again going to quote the great physicist (and equally great philosopher of science) Richard Feynman: “If you make a theory…you must also put down all the facts that disagree with it….the idea is to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.”
That is exactly what the climate science “experts” have avoided doing for the past twenty years or so.
Even if everything turns up roses, there are a whole lot of people who have staked their $$ and credibility on AGW, of which the majority of the main stream media. Considering the MSM will be trying tooth and nail to avoid an embarrassment of epic proportions, expect the inertia on this boulder of truth to be too great….for now anyways.
And not a whimper out of the Drive-By Media.
Hans, don’t worry I still work with Fortran.
,
This file is probably the most useful in showing the potential problems with temperature data. The paucity of documentation for the data files and their transformations is completely unacceptable.
I started programming in FORTRAN 40 years ago and, when I was doing research, I created some pretty bad code which I absolutely cannot comprehend when I look at it now. Even a few years after I threw together a large program over a couple of days it would be easier to write a new one rather than trying to figure out what I’d done after being awake for 24 hours straight coding.
One thing which I _always_ did was document in excruciating detail what the format of the data files was. Programs are easy to write but data from experiments is unique. Every experimental data file had reams of text indicating if I’d changed the format of the data file (which would happen often as the experiments were concurrent with major software development occurring) and exactly what was happening with the data as the experiment was in progress. The idea was that anyone coming in to look at the data would know exactly what had transpired and could write their own code to analyze the results if they didn’t like the programs I’d written. This is what eventually ended up happening when I left research to go into medical school. I got a lot of flak about my coding style but none about how I’d documented the data.
Such lack of documentation of data files is completely unacceptable. Given the potential effect of the results of CRU analyses on public policy, this omission may also be criminal in nature.
And yet:
http://news.yahoo.com/s/ap/20091123/ap_on_sc/sci_climate_09_post_kyoto
jcl … wow, talk about whistling past the graveyard, eh?
Please, everyone, post this to your Facebook sites and links on your blogs. If we keep hammering it out there, people will have to listen.
I’m going to start sending all the links to my MP in Canada and my Senator and Representative in the US congress.
In this morning’s FoxNews interview between Chris Horner at the Competitive Enterprise Institute and Howard Gould of the Clean Economy network…
In this interview, I heard Chris Horner say “there’s 62mb released and 100mb to come”. Has anyone come across this claim elsewhere, or does he have privileged information? – or did he just misspeak?
A good organization specs everything out – data, code, and processes – ahead of time and peer reviews (ha!) and validates them before a single key is pressed. Same with any and all changes.
Check out how many times the guy writes “I wrote a program.” These guys were flying by the seats of their pants.
Is there ever going to be a movie made about this scam starring all the celebrities who drive electric cars?
I’d like to up that
‘shout out’
Idd and Indiana Homez.
Thanks for the link to yet another Acorn rot Fred. Talk about fires to put out – the left- wing bedwetters will be busy calling 911 whilst dodging flying Polar Bears!
OOPS!! Was that a Polar Bear or M. Moore?? If it looks like a duck……
A good day, Indeed.
Mississauga Matt writes:
A good organization specs everything out – data, code, and processes – ahead of time and peer reviews (ha!) and validates them before a single key is pressed. Same with any and all changes.
Matt, this indicates you’ve never worked in a research setting. Writing software was a process that took time away from doing experiments and they were done as quickly as possible. This meant usually no documentation, keeping variable names short (3 characters or preferably less) to save on typing and structured code was something that real programmers didn’t do. My code was a maze of GOTO’s and computed GOTO’s and once it worked that was the end of the process and there was no reason to document as I remembered what I’d done. I was happiest coding in assembler and my FORTRAN code was setup to ensure that the memory requirements were as small as possible as everything had to fit into a 50 Kb memory space.
These programs were for use of the researchers in the lab I worked in only. I documented my assembly language code quite extensively as this was something I was proud of and would share as I think it was as optimal as one could get on a PDP-11 and the hardware configuration we were running.
When I left the lab the next programmer that came in rewrote all of the programs as he couldn’t figure out my code. He was a structured programming type of guy and it took him weeks to do what I’d done in hours. The rewrite would have been necessary even if I’d stayed in the lab as the code was ported first to a Sun workstation and later to IBM PC’s. There was far too much hardware specific stuff in my code (display on VT55 or VT100 terminals) to allow reuse of the code anyway.
Imagine the hue and cry from the left had this sort of behaviour occurred at a pharmaceutical company in an attempt to market a drug.
The company would have been, rightly, thoroughly investigated, its stocks would have plummeted and it would have very likely faced bankruptcy as all its products were withdrawn from the shelves pending the outcome of the investigation.
But hey, I guess snake oil isn’t covered by consumer protection laws.
jcl, I’ve noticed several soppy, heart-string pullers in the media in the last week or so. In some of them, the exploitation is so blatantly obvious I can’t help think there must be a planned campaign by some (probably UN supported) agency to turn the Copenhagen delegates into simpering, weeping piles of jello. These are very interesting times. When the propaganda machinery stoops to such in your face unveiled tear-jerking, you just know they have lost.
Agreed loki.
I’m not a programmer but there were a few prerequisite programming courses I needed for school because I’m heavily involved in CAD.
If a program isn’t documented line-by-line then it’s garbage, even if it’s working. Only the fact the programmers didn’t have a paying customer to whom they’re accountable enabled them get away with this. A paying customer would never accept this type of garbage.
btw, I dislike programming very much, although I feel someone fluent in programming languages should be able to claim multi-lingual status.
Dr. Tim Ball provides a well-written, inciteful comment at the Climateaudit mirror website under the topic “Curry: On the credibility of climate research”. It is located about 20 comments from the top. Well worth the read.
Embedded is this juicy tidbit about the documentary “The Greenhouse Conspiracy”:
“The movie was rejected by PBS in the US on the grounds it was biased. In Canada I understand David Suzuki kept it off the CBC”.
See: http://camirror.wordpress.com/2009/11/22/curry-on-the-credibility-of-climate-research/
Nuff said!
Indy,
As they say “if the code and the comments don’t match, they’re probably both wrong….”.
HAAAAAAAAA-HAHAHAHAHAHAHAHAHAHA!!!! This just keeps getting better and better 🙂
btw, I believe the term is “programming etiquette” wrt proper programming style and documentation.
I’m really comfortable about the idea of giving these C students absolute control of the global economy. Why not also elect a social worker as president and give him absolute control of the U.S. military?
These are wonderful ideas that will bring about a golden age of peace and prosperity — in China.
Ratt, Dr D., loki,
Yes, I remember doing machine language and Fortran. How did these “researchers” make it past 100 level comp sci?
The data sets and programs look to be completely “knackered”.
Maybe we can be saved by a “RATT”, because this viper’s nest of data poison probably needs some serious anti-venom medication.
But its probably too late to resurrect the AGW theory as it got consumed in Dante’s GLOBAL WARMING GATE INFERNO above…
Cheers
Hans-Christian Georg Rupprecht, Commander in Chief
1st Saint Nicolaas Army
Army Group “True North”
How is the AGW cultists, the MSM and the political world dealing with these revelations?
They are ignoring it. Totally ignoring it. Business as usual. Quebec has just announced a ‘massive attack on greenhouse gases’; remember, Quebecers will adopt any policy that might, just might, bring them federal funds…which they can then use as they wish for other projects.
Copenhagen continues. CNN talks about the millions lost due to rising sea levels. Only FOX has referred to it. The Blogs are filled with the scam…but the ‘Other World’ ignores it.
“My code was a maze of GOTO’s and computed GOTO’s” a.k.a. “job security”