SoftestPawn’s Weblog

Oh no, not another blog!

Posts Tagged ‘Research’

Rigourless Research Revealed

Posted by softestpawn on December 15, 2009

The argument seems to go like this: Scientists work by carefully testing their hypothesis with experiments. They test other Scientists’ work by replicating it.  This careful analysis of the Real World against theory ensures that theory is modified to fit the facts, not the other way around. They examine each others work in an open and professional and largely ego-free environment, welcoming criticism to build a better model, where those who introduce new paradigms will boost their careers.  And so we have a continuous pressure to improve theories, to adjust them and tweak them, and those changes are celebrated as progress towards a better, more detailed understanding of the World.

With this, it’s fairly obvious that Scientists in their specialist fields are very probably more right than the enthusiastic amateur layman, who does not and can not do these things without the same access to resources. And so it is right that we trust Scientists; they may be wrong, but they are more likely to be more right more of the time.

This is all fine.

Well, nearly fine

It’s fine when that definition of how Scientists work is fine.

When work cannot be tested, or is not independently replicated, then all we’re left with is the empty assertion that the particular Scientists involved are better than anyone else. In fact we know they could even be worse, since they have a motive to carry on directing results to encourage funding.

In case you forget, scientists working for tobacco companies did this not too long ago. There’s no evidence that government-funded scientists would be any different, and there is recent famous evidence that they are not. Scientists are humans too shock horror.

It also fails when the Scientist is opinionating outside their field. ‘We’ the public tend to lump ‘them’ together when we hear that ‘scientists proclaim this, that or the other’, but a medical doctor making claims about the veracity of climate has as much authority as a climatologist making claims about medicine.

Even the famous scourge of the anti-science homepath, Ben Goldacre, has forgotten the basic requirements of Good Science. He seems to simply assume, in a public newspaper, outwith his own field of expertise, that some vague concept of ‘global warming’ is true, and that the more frothy arguments from the disbelievers somehow means he can forget all his previous enthusiasm for rigour, testing, and demonstrated repeatability.

Some ‘zombie arguments’ keep resurrecting because there’s been no decent argument against them. Some (“the planet is too big for us to affect”) are just a bit wierd

So what’s wrong with that CRU research?

It’s simply not trustworthy. That doesn’t mean the people are unusually untrustworthy, or corrupt, or evil. Or even particularly incompetent, at the science at least.

There is plenty of ‘juicy gossip’ in the released emails (searchable access here), but most of it is simply that. Motivation and emotion and everyday mistakes are simply human.

There’s more than that though; there are specific examples of why we simply can’t trust the results.

Harry’s infamous readme shows just how poor the archiving of methods and data and adjustments to that data are. Where do the adjustments come from? Are they still valid? Have they been applied too often? To what? Even they don’t know.

Then there’s the approach to inconvenient results. To immediately look for reasons why the data doesn’t fit the theory, and to look for things to remove (such as ENSO, which according to that email might not even be a valid thing to remove anyway). Or indeed to hide those inconvenient results in the IPCC reports, despite protests from the data producer Keith Briffa, who protests that working to prove a consensus is not a Good Thing.

Now if there’s a mismatch between a well established theory and the data, it is natural to look for what is wrong with the data rather than the theory. You’ve measured the acceleration of your free-falling piano at more than 10m/s²? You’d better go and check your accelerometer.

But there’s a difference between checking your measurements, and adjusting your data to fit or simply ignoring the mismatches.

This is known as ‘confirmation bias‘; it’s recognised, understood, and methods for mitigating it are applied in any truly reputable working environment. There’s no evidence of such mitigation at CRU, and that’s after looking.

There are no properly independant replications. The data and some methods are shared with the GISS team (it’s not clear how much, they probably don’t either if they can’t keep their own data in order), and the satellites are calibrated from the ground station measurements. This is not some deliberate conspiracy, this is the nature of the problem, but it still means there are no properly independent replications.

The small-p political power plays –  to lean on editors to refuse publication to rivals – are not directly to do with rigour, but show quite how hard the group intends to defend their theory rather than the validity of their data.  Their vested interests (the old page listing funders has gone now, but the google cache is here) includes acknowledgements to funders including Greenpeace and WWF and many others who are quite open about their ‘vested green interests’.

It’s not just the ‘deniers’

Judy Curry, a ‘proper’ establishment climatologist,  writes on climatologist credibalityEduardo Zorita, a ‘proper’ establishment climatologist, declared the CRU researchers unfit and their work not credible (the original page has moved probably to his new blog somewhere, there’s a copy here with a rather enthusiastic skeptic).

Even George Monbiot, a keen alarmist, feels quite rightly that if the alarmists are to have a case they must make sure they ‘uphold the highest standards of science’, though he obviously isn’t ready to face the consequences of what it means if they haven’t.

So where does this leave us?

There is no sensible room for deliberate worldwide conspiracies of hoaxes or scams, but only the more frothy deluders claim that. That’s a convenient “He’s a twonk, so you must be all be twonks, therefore I’m right” argument by alarmists (just as it’s raised by deniers over, say, the affiliation of Al Gore).

Incompetent Systems™ still apply: incentives to encourage behaviour that doesn’t manufacture data, but tweaks it just a little, adjusts it a little more, finds plausible factors to remove. No checks to ensure that rigour is applied, well, rigorously. There is plenty of hard work directed towards proving a theory, not testing it; but as long as we assume general professional honesty, then it will all be within fairly limited boundaries.

It’s unlikely we’re going to get a very different result when – eventually, one day – it’s properly and thoroughly analysed.

Even so, results that, say, indicate there was a medieval warming period that is warmer than today would cause the alarmists some trouble getting their point across. Since they’ve sold their cause as ‘unprecedented’ global warming, a large part of their point would disappear. Which isn’t at all helpful if they’ve got enough of the science accidentally right and the world is in significant peril.

That’s a reason to go back and do the science properly right, as soon as possible, much sooner rather than a bit later, and definitely not let the researchers back into their holes crying “It’s not fair that you caught me this time! I’m right anyway, everyone I know says so!”

Posted in Environmentalism, Global Warming, Science | Tagged: , , | 8 Comments »

Peer Review in the Real World

Posted by softestpawn on August 10, 2008

Some academics seem to think peer review is a suitable way to establish knowledge, with a nod to “it’s not perfect”. But imagine applying it in the real world:

“We need to know more about Wodgets to feed the Widget project”

“OK, so what’s the plan for the investigation?”

“We’ll I’m going to get the team together, explain the problem and let them get on with it”

“Heh. So, how are you going to monitor progress?”

“Oh we’ll have a regular review document where they can publish what they’ve found out. It’ll need to be fairly brief, we don’t want everyone wasting time reading about everyone else’s work. We’ll get colleagues in the same field to select them by how interesting they are, but I don’t think we need to enforce any other requirements. It’ll take too long to see if it’s correct for example, no point in duplicating effort. And we’ll give credits to those who get articles published that will contribute to promotion. They’re all professionals, it won’t affect the areas they work in, the results, or team relationships at all”.

“Heh. Ha ha. What then?”

“Oh nothing extra. We’ll just keep doing that. If people read the review documents they might get around to critically examining some of the articles, but there’s no real need as it’ll get in the way of their work. If someone does happen to find something wrong, they can write an article for the next review document, and if it gets published then everyone who happens to read it will know about the correction. If they notice this affects some other work they can update that when they get around to it, and see about publishing that correction and so on”

“Ha ha ha ” (Collapses in gales of laughter). “And how will we know when we’ve established a reasonable knowledge base of Wodgets?”

“Ah when experts – those who have lots of credits – feel that the articles in the journals all say roughly the same thing, they’ll tell us that they all agree with how a Wodget works. We might see if some researchers on the Wadget project can drop in occasionally and agree with them.”

“Hoo brilliant! That’s fantastic!” (wipes away tears, snot and the coffee sprayed over the table) “Anyway what are you really going to do?”

“Follow the department’s standard quality process … ”

“Oh how dull. Well carry on. Keep me informed as usual”

—–

Postscript: Academics are working on it

Posted in Science, Silliness | Tagged: , , , | Leave a Comment »