SoftestPawn’s Weblog

Oh no, not another blog!

Rigourless Research Revealed

Posted by softestpawn on December 15, 2009

The argument seems to go like this: Scientists work by carefully testing their hypothesis with experiments. They test other Scientists’ work by replicating it.  This careful analysis of the Real World against theory ensures that theory is modified to fit the facts, not the other way around. They examine each others work in an open and professional and largely ego-free environment, welcoming criticism to build a better model, where those who introduce new paradigms will boost their careers.  And so we have a continuous pressure to improve theories, to adjust them and tweak them, and those changes are celebrated as progress towards a better, more detailed understanding of the World.

With this, it’s fairly obvious that Scientists in their specialist fields are very probably more right than the enthusiastic amateur layman, who does not and can not do these things without the same access to resources. And so it is right that we trust Scientists; they may be wrong, but they are more likely to be more right more of the time.

This is all fine.

Well, nearly fine

It’s fine when that definition of how Scientists work is fine.

When work cannot be tested, or is not independently replicated, then all we’re left with is the empty assertion that the particular Scientists involved are better than anyone else. In fact we know they could even be worse, since they have a motive to carry on directing results to encourage funding.

In case you forget, scientists working for tobacco companies did this not too long ago. There’s no evidence that government-funded scientists would be any different, and there is recent famous evidence that they are not. Scientists are humans too shock horror.

It also fails when the Scientist is opinionating outside their field. ‘We’ the public tend to lump ‘them’ together when we hear that ‘scientists proclaim this, that or the other’, but a medical doctor making claims about the veracity of climate has as much authority as a climatologist making claims about medicine.

Even the famous scourge of the anti-science homepath, Ben Goldacre, has forgotten the basic requirements of Good Science. He seems to simply assume, in a public newspaper, outwith his own field of expertise, that some vague concept of ‘global warming’ is true, and that the more frothy arguments from the disbelievers somehow means he can forget all his previous enthusiasm for rigour, testing, and demonstrated repeatability.

Some ‘zombie arguments’ keep resurrecting because there’s been no decent argument against them. Some (“the planet is too big for us to affect”) are just a bit wierd

So what’s wrong with that CRU research?

It’s simply not trustworthy. That doesn’t mean the people are unusually untrustworthy, or corrupt, or evil. Or even particularly incompetent, at the science at least.

There is plenty of ‘juicy gossip’ in the released emails (searchable access here), but most of it is simply that. Motivation and emotion and everyday mistakes are simply human.

There’s more than that though; there are specific examples of why we simply can’t trust the results.

Harry’s infamous readme shows just how poor the archiving of methods and data and adjustments to that data are. Where do the adjustments come from? Are they still valid? Have they been applied too often? To what? Even they don’t know.

Then there’s the approach to inconvenient results. To immediately look for reasons why the data doesn’t fit the theory, and to look for things to remove (such as ENSO, which according to that email might not even be a valid thing to remove anyway). Or indeed to hide those inconvenient results in the IPCC reports, despite protests from the data producer Keith Briffa, who protests that working to prove a consensus is not a Good Thing.

Now if there’s a mismatch between a well established theory and the data, it is natural to look for what is wrong with the data rather than the theory. You’ve measured the acceleration of your free-falling piano at more than 10m/s²? You’d better go and check your accelerometer.

But there’s a difference between checking your measurements, and adjusting your data to fit or simply ignoring the mismatches.

This is known as ‘confirmation bias‘; it’s recognised, understood, and methods for mitigating it are applied in any truly reputable working environment. There’s no evidence of such mitigation at CRU, and that’s after looking.

There are no properly independant replications. The data and some methods are shared with the GISS team (it’s not clear how much, they probably don’t either if they can’t keep their own data in order), and the satellites are calibrated from the ground station measurements. This is not some deliberate conspiracy, this is the nature of the problem, but it still means there are no properly independent replications.

The small-p political power plays –  to lean on editors to refuse publication to rivals – are not directly to do with rigour, but show quite how hard the group intends to defend their theory rather than the validity of their data.  Their vested interests (the old page listing funders has gone now, but the google cache is here) includes acknowledgements to funders including Greenpeace and WWF and many others who are quite open about their ‘vested green interests’.

It’s not just the ‘deniers’

Judy Curry, a ‘proper’ establishment climatologist,  writes on climatologist credibalityEduardo Zorita, a ‘proper’ establishment climatologist, declared the CRU researchers unfit and their work not credible (the original page has moved probably to his new blog somewhere, there’s a copy here with a rather enthusiastic skeptic).

Even George Monbiot, a keen alarmist, feels quite rightly that if the alarmists are to have a case they must make sure they ‘uphold the highest standards of science’, though he obviously isn’t ready to face the consequences of what it means if they haven’t.

So where does this leave us?

There is no sensible room for deliberate worldwide conspiracies of hoaxes or scams, but only the more frothy deluders claim that. That’s a convenient “He’s a twonk, so you must be all be twonks, therefore I’m right” argument by alarmists (just as it’s raised by deniers over, say, the affiliation of Al Gore).

Incompetent Systems™ still apply: incentives to encourage behaviour that doesn’t manufacture data, but tweaks it just a little, adjusts it a little more, finds plausible factors to remove. No checks to ensure that rigour is applied, well, rigorously. There is plenty of hard work directed towards proving a theory, not testing it; but as long as we assume general professional honesty, then it will all be within fairly limited boundaries.

It’s unlikely we’re going to get a very different result when – eventually, one day – it’s properly and thoroughly analysed.

Even so, results that, say, indicate there was a medieval warming period that is warmer than today would cause the alarmists some trouble getting their point across. Since they’ve sold their cause as ‘unprecedented’ global warming, a large part of their point would disappear. Which isn’t at all helpful if they’ve got enough of the science accidentally right and the world is in significant peril.

That’s a reason to go back and do the science properly right, as soon as possible, much sooner rather than a bit later, and definitely not let the researchers back into their holes crying “It’s not fair that you caught me this time! I’m right anyway, everyone I know says so!”


8 Responses to “Rigourless Research Revealed”

  1. I would like to point out also that Anthony Watt’s surface station project calls into question whether NOAA is measuring global warming or the development of urban heat islands. Unlike GISS or CRU, Anthony Watt has posted his research results on the web for all to see. What we see is that in the U. S. At least, we have good reason to suspect that the majority of surface temperature measurements have been compromised. (For those who have not actually visited his Web site, the photographs of the measuring stations are posted) . It is therefore critical, in terms of assessing the accuracy of the data, to know what adjustments were made to measured temperatures (the raw data) and why. Yet it is that very information that is being withheld by the so-called experts. That isn’t science, it’s chicanery.

  2. timheyes said

    At last. An incontrovertable argument proving AGW is a load of rubbish. Thank God this guy is on our side…


  3. Karen said

    This is the first time I’ve visited your site and I’ve only skimmed through your previous climate change postings, but don’t think you’ve mentioned this piece – Climate Change and the Death of Science. ( )

    According to Professor of Climate Change, UEA’s Mike Hulme, climate change isn’t supposed to be hard science, dealing with facts, but a ‘key narrative’, ‘an intellectual resource’, ‘a resource of the imagination’, ‘an imaginative condition of human existence’ – I won’t go on because you’ve quite possibly already read it, or perhaps the whole subject of ‘post-normal’ science is now old hat in the scientific community.

    I’m utterly crap at science, so perhaps my faith in scientists as disinterested seekers-after-truth is a bit naive. I just found the whole thing quite shocking.

    • Interesting article, thanks.

      I don’t think “science” (or rather, academic research) as a whole is broken, but like anything else it gets a bit mangled when it gets political. It’s always been that way, and indeed I think recently things like better objective methods and public scrutiny are improving it.

      We can still spot and poke at and try and separate the political/financial incentives from the hard facts, without necessarily understanding the science.

  4. Karen said

    Thanks for your response.

    I appreciate that scientific endeavour is often framed within an agenda that is less than pure; Big Tobacco is usually mentioned as an example of this, although ‘Big Pharma’ seems to be catching up fast! Follow the money, as they say.

    What I find so unsettling about the ‘Death of Science’ article is that this Ravetz chap appears to be making a serious attempt to legitimise what amounts to scientific fraud. It’s an ends-justify-the-means approach and that just seems WRONG! It doesn’t matter how white the lies are.

    It makes you wonder what kind of science is implied by the phrase, ‘the science is settled’. The post-normal variety certainly appears to be settled before it even begins.

    Finally – and I don’t want to hog your thread! – I’m not sure what you mean by, ‘… things like better objective methods and public scrutiny are improving’ academic research. For public scrutiny, is this a reference to the introduction of FOI legislation? In which case, the behaviour of the researchers at CRU shows how easily rules can be bent. The peer review process likewise. Widespread internet use has certainly played its part – without it, Woolanders (like me), would never have had access to any of this.

    Not sure if that’s a good or bad thing.

    • You’re welcome! I think there’s very little ‘science’ being done when the ends justify the means, but that’s just politics-pretending-to-science rather than science, and we can often recognise it as such.

      I feel that things are improving because the old close research communities are breaking open, systematic reviews such as Cochraine’s are being refined and increasingly widely understood, research institutions introducing quality practices such as full disclosure and data auditing, and wide public scrutiny are putting pressure on the IPCC for example to be more rigorous.

      Yes people still and will try and get away with poor practice, but it’s increasingly hard to do so. I think. I hope!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: