Debunking the Debunking Handbook

SkepticalScience has published the Debunking Handbook that is intended to summarise how you show an argument is wrong. Unfortunately… it’s wrong in some fairly fundamental ways.

The summary at the beginning says:

“Debunking myths is problematic. Unless great care is taken, any effort to debunk misinformation can inadvertently reinforce the very myths one seeks to correct. To avoid these “backfire effects”, an effective debunking requires three major elements. First, the refutation must focus on core facts rather than the myth to avoid the misinformation becoming more familiar. Second, any mention of a myth should be preceded by explicit warnings to notify the reader that the upcoming information is
false. Finally, the refutation should include an alternative explanation that accounts for important qualities in the original misinformation”

For a start this is phrased to suggest that you already know what is fact and what is ‘myth’. ie, this is not a way of evaluating an argument for its worth or otherwise, but a way of selling a specific argument.

It’s a ‘spin’ manual.

First: the refutation must focus on core facts rather than the myth

So you push your own facts and avoid considering in detail those that contradict them. Any non-trivial problem has myriads of facts that can be interpreted in different ways to suggest different conclusions – this is what makes understanding people, the world and the universe so interesting. This is far from ‘debunking’ an argument; to focus on specific facts and avoid others is spin.

The example given is the claim that (some) climate skeptics claim that the sun has driven recent climate warming. The debunking is supposedly that the sun’s measured total radiation output does not match warming in the last very few decades, and therefore the skeptic claim is wrong. By itself, this is Fine and Good, but ignores the myriad effects that various solar outputs – different particles and radiation wavelengths – have on the atmosphere and so temperatures. The conclusion may well be right, but the text ignores or oversimplifies the facts that support an alternate view and picks those that support the agenda of the so-called ‘debunker’. This is not a debunk, it’s a sell.

Second: any mention of a myth should be preceded by explicit warnings

This is an obvious statement of intent: a claim that the argument is wrong without saying why.

It’s not even a refutation, let alone a debunk.

Finally: the refutation should include an alternative explanation

This is clearly wrong as it has nothing to do with showing how the initial argument is wrong, and can result in missing the point.

If you claim that aliens move clouds around, I can counter with a similarly clueless argument that the clouds are sentient and move themselves. The discussion can then move to how silly it is that clouds are sentient, and so lose the focus from evaluating the original claim about aliens.

At worst, having shown that it is silly to think that clouds are sentient, a (poor) conclusion is that therefore aliens do indeed move clouds around, as the only alternative considered.

The Worldview Backfire Effect

It is ironic that such a publication should talk about how people are biased by their “worldviews and sense of cultural identity” without considering how they might affect the authors.

In particular I enjoyed the phrase “Self-affirmation and framing aren’t about manipulating people” because, clearly, they are (see also, for example, the very interesting article by Kahneman and Tversky Judgements Under Uncertainty). That’s what is interesting about them.

Removing framing to get at the underlying objective data and arguments is extremely difficult, and will continue to be so while publications like the “Debunking Handbook” encourage others to muddy the waters.

Aggregating Adversarial Argument

I am reasonably intelligent. I am interested in the topic, and so am reasonably well read and informed on it. The conclusions I come to follow reasoned lines, match presented evidence well, and are rationally the most likely. They are scientific. Many other interested, intelligent people have also come to the same conclusions.

You, however, disagree. How can this be? Since my conclusions are rational and informed, you must be biased by your ideology, your vested interests. You must be selectively ignoring key evidence and denying the science; you must therefore be ‘anti-science’. Maybe you have been persuaded by misinformation distributed by vested interests and lobbyists, using tried and tested techniques to appeal to your emotions rather than reason. Or maybe you just lack the mental skills to be able to properly assess the complexities and uncertainties.

Convinced? No? Then it must be because one ‘cannot reason someone out of a belief they did not reason themselves into’. I am still the rational, correct one, and you are, basically unreasonable.

Yeah, right.

This inability to understand, assess, value and maybe even argue convincingly for wide-spread opinion that we disagree with is a sure sign of intellectual weakness in an argument. It suggests ideologically-based bias: we have been too narrowly selective about the ways in which we assess the facts, the reasons and the effects. Can those of us interested in these things make good cases for both evolution and creationism? For homoeopathy? Both for and against late term abortion? Any abortion? For cake? Smoking in pubs? Permitting a new local Tesco’s? How much fruit and veg we ‘should’ eat? And for what?

If we cannot understand and adjust the priorities assigned to evidence and reason that is required to come to the conclusions that many other people hold, then we probably haven’t understood the problem properly. Even if later, with hindsight, we find we have come to the ‘correct’ conclusion, it is likely by accident or social identity, not reason.

This is not about persuading or convincing, about getting inside the heads of people you disagree with in order to be able to change their mind. It’s about better understanding the problems and their associated issues, and so coming to conclusions that better usefully match the real world.

Visualising the Remote Sensing Satellite data

To practice getting back into coding again, I wrote a quick visualiser for the global temperature satellite measurements, and it’s quite pretty so seems worth presenting.

The RSS data is already presented as graphs and outlines here, but I thought it would be interesting to see what happened across the globe over time.

The data files for the TLT (near surface ‘brightness’) from here: ftp://ftp.ssmi.com/msu/data/binary/

I plotted these using Java into four panels:

• top left simply shows brightness corresponding to the absolute temperature
• top right is the anomaly according to the average data set. ie, each pixel was averaged over the time series, then for each time step the difference is shown. Negatives are blue, positives red.
• Bottom left is the anomaly averaged over 12 months to remove seasonal effects. ie, each pixel was averaged as above across the whole series, then again for 6 months previous and subsequent to the time step, and the difference shown. Negatives are blue, positives red.
• Bottom right is as above but over 24 months.

A typical frame looks like this:

Lots of not rights here: I’m not quite sure which months correspond to exactly which year, I’ve not got the colour distributions right so there are occasional wierd pixels, and the shape isn’t right.

It’s still quite pretty; you can see the seasonal shifts from north to south hemisphere, and the outline of the continents on the absolute temperature frame top left. The bottom two show the changing patterns of local weather over the medium term.

I assembled a subselection into an animated gif; every 3rd month and about half (1994-2007ish) the full time range (1985-2011) . This is a subset as my GIF animator only handles something like 70 frames (which is fair enough, should really find out how to do this as a proper video) which hopefully will show here, though it’s an 11MB file:

Code is available if you want it.

Used a trial version of A4Video (looks useful) to assemble this video for most of the time range (it’s top-and-tailed a bit):

BBC’s (Im)partial Science Reporting

The BBC is holding another review on its impartiality, this time on how it presents scientific subjects: Science impartiality review – terms of reference (PDF). It has existing guidelines, and it has held such reviews before on how it reports on subjects such as religion and the middle east. This is all Good Stuff, as the BBC’s reputation rests somewhat on the quality and reliability of its reporting, and reliability requires, among other things, impartial reporting. 

One of the many frustrations for medically trained scientists however is the airtime and article space given to ‘alternative’ treatments such as homeopathy, reiki, accupuncture and so on. These are treatments that have not passed the objective tests used to identify those that actually work. These tests (double blinded, randomised control groups, etc) are meant to bypass the personal and social prejudices and biases that affect our abilities to properly evaluate effectiveness. They do not always succeed.

The concern is largely that by giving publicity to unproved, useless and sometimes dangerous treatments, the BBC lends them credibility and authority, and so more people may be taken in by them.  By providing BBC publicity to such sites as JABS, people may believe them to be officially sanctioned.

And so these concerned people do not want the BBC to give equal space to these cranks, charlatans and quacks. Such reporting is not truly balanced, they claim. If you’re going to report science, they say, you should report scientific science not pseudoscience.

Scientific science vs pseudoscience

Which all sounds well and good, but the BBC does not have the funds or indeed the expertise to properly evaluate every controversial issue.  For a start, only a few controversies can be tested in the clearly objective way that medical treatments can. 

The BBC may instead decide to defer all evaluation to certain establishment scientists and report only the expert opinions of people with certain qualifications from certain institutions; but this is not scientific. It’s not uncommen for academic research scientists to fall prey to their own or others pseudoscience, even in related fields.

Nor does the BBC have the remit to make such evaluations or deferrals. A public controversy is one with many people who believe opposing things, for frequently unscientific reasons, and the BBC’s audience is public. If the BBC were to fail to report the views of such people and how they were derived, then it is failing to engage with or inform the discussion.

The concerned may argue that such a discussion is not a scientific one: a programme on ghosts has no place under the Science label for example. Yet the evaluation of sparse evidence is vital to science; a negative result is still a useful result. And we need not be sheltered from uncertain and ambiguous evidence, leaving us to make up our own minds - this too, is science. 

Impartial to the audience, not the evidence

Impartiality is not the same as correctness. The BBC can and should provide time to the different parties in a discussion that the general public is interested in.

This doesn’t mean having to give airtime to any old crackpot view, but if large proportions of the public are, say, worried about vaccinations then it is quite right of the BBC to air those concerns along with objective evaluations of them. The BBC rightly provides a platform for those advocates to present their case to the public, for the public to evaluate. 

The public – everyone – is indeed ignorant and stooopid about most subjects (who has time to evaluate everything?). But to be protected from our own folly and expertise by filtering what is presented to us leaves us in the hands – and frequently inexpert opinions – of those doing the filtering. 

So yes, let’s have links to sources so we can check back and do our own evaluations. Let’s have more entertaining educating articles and programmes such as those from More-or-Less and Ben Goldacre. And let’s have more time to hear the cases rather than have them forced into small soundbites. 

But let’s not start letting partisan groups decide on our behalf what we should hear about when it comes to science topics. Because that’s really not good scientific practice.

From Stuff and Nonsense &  DC’s Improbable Science, although the review started back in March. A cutdown version of this has been sent to the BBC’s feedback email: trust.science@bbc.co.uk

Not “Winning the War on…”

“The war on” terrorism, drugs, crime, poverty and so on frames these activities as a struggle with an end: either victory or defeat. As with the Second World War, the armies will destroy the enemy’s will to fight, the victors will introduce a Marshall Plan to fix everything that got broken, and then everyone settles down to enjoy peace and posterity.

If the end does not arrive nice and promptly, that the war is going on ‘too long’. The activity may not even be winnable. And if it’s not winnable, then we should give it up as pointless.

This metaphor seems suitable when ‘The War On..’ involves real war. The invasion of Iraq and the struggle for control of areas of Afghanistan had and have soldiers and gun battles and people deliberately trying to kill each other.

But real actual killing warfighting is only a fairly specific component of even a military-backed mission. Without other activities the overall intent can fail: the coalitions plainly won the war to control Iraq. They just never quite established that control themselves, and what they did they lost – through ‘factors other than war’. Even within modern armed conflicts we rarely operate with the clear remits for killing that we associate with clear warfare. The rules of engagement in Afghanistan are heavily restricted to reduce casualties amongst civilians; this necessarily prolongs the fire fights but is intended to improve the overall situation.

The wars on crime, on drugs, on poverty are more obviously foolish as useful metaphors.  There is no expected end state on any of these; we don’t expect crime to ever disappear, and definitions of poverty tends to change as our overall standard of living improves.

We could just give up the struggle; ‘the war on drugs’ could be ended (in defeat? victory?) by legalising all drugs. While it solves a few issues, it doesn’t suddenly solve debilating addictions. Some activities-other-than-war would still be needed.

Using metaphors like ‘the war on’ doesn’t just hide all these other important activities, it makes it difficult to appreciate them and balance them, and it provides the wrong references for judging progress.

‘Fighting crime’ for example involves quite a lot of not fighting at all: improving education to improve prospects, encouraging employment, changes to social and community attitudes.

A much more appropriate metaphor is gardening. There is no end-state ‘win’ in gardening; there are desired states for the garden or parts of it but work does not stop when you reach it. Indeed, some gardens require tremendous effort to maintain. You can change your intended result to make the work easier or harder,  but you have to keep working to maintain an approximation to that result.

And while “Crime Gardening” doesn’t really cut the spin mustard, some metaphors might be usable: Weeding Out Crime, Landscaping A New Society, Cultivating Good Relationships, and probably something about Roots.