Rigourless Research Revealed
Posted by softestpawn on December 15, 2009
The argument seems to go like this: Scientists work by carefully testing their hypothesis with experiments. They test other Scientists’ work by replicating it. This careful analysis of the Real World against theory ensures that theory is modified to fit the facts, not the other way around. They examine each others work in an open and professional and largely ego-free environment, welcoming criticism to build a better model, where those who introduce new paradigms will boost their careers. And so we have a continuous pressure to improve theories, to adjust them and tweak them, and those changes are celebrated as progress towards a better, more detailed understanding of the World.
With this, it’s fairly obvious that Scientists in their specialist fields are very probably more right than the enthusiastic amateur layman, who does not and can not do these things without the same access to resources. And so it is right that we trust Scientists; they may be wrong, but they are more likely to be more right more of the time.
This is all fine.
Well, nearly fine
It’s fine when that definition of how Scientists work is fine.
When work cannot be tested, or is not independently replicated, then all we’re left with is the empty assertion that the particular Scientists involved are better than anyone else. In fact we know they could even be worse, since they have a motive to carry on directing results to encourage funding.
In case you forget, scientists working for tobacco companies did this not too long ago. There’s no evidence that government-funded scientists would be any different, and there is recent famous evidence that they are not. Scientists are humans too shock horror.
It also fails when the Scientist is opinionating outside their field. ‘We’ the public tend to lump ‘them’ together when we hear that ‘scientists proclaim this, that or the other’, but a medical doctor making claims about the veracity of climate has as much authority as a climatologist making claims about medicine.
Even the famous scourge of the anti-science homepath, Ben Goldacre, has forgotten the basic requirements of Good Science. He seems to simply assume, in a public newspaper, outwith his own field of expertise, that some vague concept of ‘global warming’ is true, and that the more frothy arguments from the disbelievers somehow means he can forget all his previous enthusiasm for rigour, testing, and demonstrated repeatability.
Some ‘zombie arguments’ keep resurrecting because there’s been no decent argument against them. Some (“the planet is too big for us to affect”) are just a bit wierd
So what’s wrong with that CRU research?
It’s simply not trustworthy. That doesn’t mean the people are unusually untrustworthy, or corrupt, or evil. Or even particularly incompetent, at the science at least.
There is plenty of ‘juicy gossip’ in the released emails (searchable access here), but most of it is simply that. Motivation and emotion and everyday mistakes are simply human.
There’s more than that though; there are specific examples of why we simply can’t trust the results.
Harry’s infamous readme shows just how poor the archiving of methods and data and adjustments to that data are. Where do the adjustments come from? Are they still valid? Have they been applied too often? To what? Even they don’t know.
Then there’s the approach to inconvenient results. To immediately look for reasons why the data doesn’t fit the theory, and to look for things to remove (such as ENSO, which according to that email might not even be a valid thing to remove anyway). Or indeed to hide those inconvenient results in the IPCC reports, despite protests from the data producer Keith Briffa, who protests that working to prove a consensus is not a Good Thing.
Now if there’s a mismatch between a well established theory and the data, it is natural to look for what is wrong with the data rather than the theory. You’ve measured the acceleration of your free-falling piano at more than 10m/s²? You’d better go and check your accelerometer.
But there’s a difference between checking your measurements, and adjusting your data to fit or simply ignoring the mismatches.
This is known as ‘confirmation bias‘; it’s recognised, understood, and methods for mitigating it are applied in any truly reputable working environment. There’s no evidence of such mitigation at CRU, and that’s after looking.
There are no properly independant replications. The data and some methods are shared with the GISS team (it’s not clear how much, they probably don’t either if they can’t keep their own data in order), and the satellites are calibrated from the ground station measurements. This is not some deliberate conspiracy, this is the nature of the problem, but it still means there are no properly independent replications.
The small-p political power plays – to lean on editors to refuse publication to rivals – are not directly to do with rigour, but show quite how hard the group intends to defend their theory rather than the validity of their data. Their vested interests (the old page listing funders has gone now, but the google cache is here) includes acknowledgements to funders including Greenpeace and WWF and many others who are quite open about their ‘vested green interests’.
It’s not just the ‘deniers’
Judy Curry, a ‘proper’ establishment climatologist, writes on climatologist credibality. Eduardo Zorita, a ‘proper’ establishment climatologist, declared the CRU researchers unfit and their work not credible (the original page has moved probably to his new blog somewhere, there’s a copy here with a rather enthusiastic skeptic).
Even George Monbiot, a keen alarmist, feels quite rightly that if the alarmists are to have a case they must make sure they ‘uphold the highest standards of science’, though he obviously isn’t ready to face the consequences of what it means if they haven’t.
So where does this leave us?
There is no sensible room for deliberate worldwide conspiracies of hoaxes or scams, but only the more frothy deluders claim that. That’s a convenient “He’s a twonk, so you must be all be twonks, therefore I’m right” argument by alarmists (just as it’s raised by deniers over, say, the affiliation of Al Gore).
Incompetent Systems™ still apply: incentives to encourage behaviour that doesn’t manufacture data, but tweaks it just a little, adjusts it a little more, finds plausible factors to remove. No checks to ensure that rigour is applied, well, rigorously. There is plenty of hard work directed towards proving a theory, not testing it; but as long as we assume general professional honesty, then it will all be within fairly limited boundaries.
It’s unlikely we’re going to get a very different result when – eventually, one day – it’s properly and thoroughly analysed.
Even so, results that, say, indicate there was a medieval warming period that is warmer than today would cause the alarmists some trouble getting their point across. Since they’ve sold their cause as ‘unprecedented’ global warming, a large part of their point would disappear. Which isn’t at all helpful if they’ve got enough of the science accidentally right and the world is in significant peril.
That’s a reason to go back and do the science properly right, as soon as possible, much sooner rather than a bit later, and definitely not let the researchers back into their holes crying “It’s not fair that you caught me this time! I’m right anyway, everyone I know says so!”