Elitism & Science

As the he-said-so-he-must-have-meant pop-psychology goes on over the unexpectedly published EA CRU data, some of the discussion turns to how scientists (or, to be more specific, academic researchers) involved should behave professionally.

We’re all – even most academic researchers – human. We can expect Phil Jones and his team to be angry, to scorn those who question his theories, especially when he sees those theories as vital to the future of humanity. And so he does. So would most of us, though we may be a little more careful about committing those things to email.

Dr Spencer, a well known skeptic, has quite a lot to say about such ‘elitist’ behaviour

Good -isms

But there’s nothing wrong with being elite, with being amongst the best at doing a job. And being able to discriminate on merit – on the ability to do things well – is a vital part of any society that intends to improve its lot.

We similarly must discriminate between ages if we want to avoid sharing a wing with Sidney Cook. We discriminate between religions to book holidays, and when providing meals to guests. We discriminate between sexes if we’re heterosexual, or homosexual. We discriminate between sexual preferences to ensure that those that can’t discriminate between ages get to share a wing with Sidney Cooke. We discriminate again between ages to allow certain ages to get away with not making that discrimination.

And this is all good, if a bit Sir Humphrey.

Bad -isms

But if we consider ‘elitism’ as we consider ‘racism’ (discriminating for differences in behaviour or ability that don’t exist), then we’ve got a much more unpleasant attitude. Then we get people who think that their superior expertise gives them remit to protect that expertise by denying evidence to others, remit to use political or organisation clout to deny them access to publish, or remit to disregard any work by anyone else purely because they are not also ‘officially’ elite.

I’m not convinced however by Spencer’s claim that the CRU team are elitist in that way. Yes they believe themselves right, they believe Spencer and McIntyre and McKintrick and all the other hundreds of skeptical scientists are wrong, and they act as the mini tribe that most of us act when we consider ourselves ‘us’ and others ‘them’. There’s nothing particularly unusual with showing they despise people they think are very wrong and are undermining their hard efforts. Even when it’s rather callous.

And there’s nothing particularly evil about abusive comments from experts about other people’s competence. These are arguments over merit, based on their opinions of each other’s work.

Ordinary tribal -ism.

Declaring that those opinions matter only when they are part of the community is not so good. Apparently they are only worth considering when published in approved journals. Journals that publish them are not approved of, and should be ousted from the community. By somewhat underhand means. Which makes for a nice, comfortable, insular, self-reinforcing community, or ‘ivory tower’ as it is usually known.

So they appear to ‘discriminate against’ McIntyre for example because he’s not part of their community, rather than because he’s not ‘elite’. His theories are ‘discredited’ because they are not published in the community journals, rather than because they are wrong.

He certainly doesn’t fit the community: he publishes openly, on t’interweb, where anyone can and does criticise his work (of course, the CRU community is also now doing this, inadvertantly, and they don’t like it). He has a background in statistics, not environment, and he generally sticks to statistical analysis. And while he’s definitely not an enthusiast for The Cause, he’s careful to remain neutral on what the final conclusion will be.

That Science Thang Agiin.

More importantly than disregarding opinion outside the community (we’re all busy anyway, how much time have we got to consider every criticism everywhere?) or the ordinary abuse and wishful thinking, there are the fairly deliberate discussions about (mis)interpreting the data to fit the cause (eg Bishop Hill, Delingpole – these include some rather dubious criticisms of the emails, but some are very telling).

The complete opposite of the much-vaunted stereotyped scientist that is curious about the differences between theory and observation, and investigates them.

Even so, if these particular twonks demonstrate poor professionalism, bordering and perhaps crossing to deliberate manipulation, misrepresentation and destruction of the data, that only means some of these folks do (some are much better behaved). It would be poor science to infer that’s the case for all climatologists, or reflects on the conclusions of the climatology community as a whole.

Though we might want to check that the wider community is more professional – more scientific – in the same way that we might want to check any other organisation for systematic incompetence when we uncover some in a core part of it.

Anyhow, a few paragraphs from Spencer’s article make much better points about how we outside these academic research communities should view the work that they do:

One of the biggest misconceptions the public has about science is that research is a straightforward process of making measurements, and then seeing whether the data support hypothesis A or B. The truth is that the interpretation of data is seldom that simple.

There are all kinds of subjective decisions that must be made along the way, and the scientist must remain vigilant that he or she is not making those decisions based upon preconceived notions. Data are almost always dirty, with errors of various kinds. Which data will be ignored? Which data will be emphasized? How will the data be processed to tease out the signal we think we see?

Hopefully, the scientist is more interested in discovering how nature really works, rather than twisting the data to support some other agenda. It took me years to develop the discipline to question every research result I got. It is really easy to be wrong in this business, and very difficult to be right.

We can see that we need to do better than ‘hope’, if we are to get any reliable science to inform our votes, lobbying and ‘lifestyles’ on this matter.

Update:Judith Curry (I think this climate researcher), talks about tribalism and the duty of public release here

Hacking: It’s Good for Science

Over the last few days the global warming communities – those ‘for’ and ‘against’ – have been deluged by the news that the computer systems at Hadley’s Climate Research Unit (part of the British Met Office*) have been hacked and the data posted on t’interwebs:

• WUWT: Hadley CRU hacked
• Climate Audit
• Real Climate (staffed by many ‘close’ to the CRU victims)

The alleged docs are here (along with on-line searchable access to the emails) but of course, this is the internet, and you can make up anything you like and post it. (Update: Downloading and expanding it, it appears to include 100mb of uncompressed code and data, mostly tree-ring/bristlecone proxy rather than weather station measurements. If this is made up, then someone’s been very busy; but there is also a danger that it is ‘mostly real’ with some key edits)

Assuming for the moment that these are real, and that Phil Jones does in fact admit it, then this is not good for the reputation of Science-The-Human-Endeavour. The tone and contents of the emails squash any claim that ‘you can trust us, we’re scientists, we’re objective and only interested in the facts’ (but then, we know that humans don’t do science)

It doesn’t even help, much, the scientific debate on global warming. As the above discussions show, the main responses are around dishonesty and legality (which are somewhat open to interpretation), rather than analysing the facts and the data. But then the scientific debate has always been very sparse across this general debate; everyone claims to have science on their side and will point to authority, to motivations, to allegiance, to politics, to vested interests, to the number of people working on it, even to assumed ideologies, in order to bolster that claim, but few will actually discuss the science. Well, the science is difficult OK?

But that will come. After the quote mining and short-term tribalist gloating is over, the Big Win for science is the simple straightforward forced releases of data that so far has been kept hidden, for possibly good but still also hidden commercial reasons (That is, CRU wouldn’t show any evidence for why it should be kept hidden, because they claimed to have lost that evidence).

Real Science – that is, the accumulation of a systematic body of knowledge, rather than the insular world of messy so-called-iterative academic research – requires rigour. It requires openness. It requires criticism, whether deserved or not, to tighten arguments and improve evidence quality, and expose gaps and risks. In other words, it requires independent review, or at the very least the threat of it.

Openness is forced internally in any organisation or project that practices ‘due diligence’. We have seen it introduced to medicine in only the last generation or so; many academic organisations** have been reluctant, slow and late to that particular party, for all kinds of ordinary people and practical reasons.

This hack – an externally forced openness – will not do much good in the short term, especially to those involved. But in the long term, we can hope to see researchers who inform public policy become openly professional – and scientific – throughout their work, because now they know that someone, internal or external, may come along one day soon and let unfriendly people examine it. All of it.

Update:Judith Curry (I think this climate researcher), talks about tribalism and the duty of public release here

—

* I’m not actually clear on the differences in responsibilities and allegiances of the Hadley center, The British Met Office, and East Anglia University’s Climate Research Unit. I don’t think they are either.

** And plenty of private organisations too. I’m just picking on researchers whose work is used to drive public policy (and I’ve made some changes to the text to make this clear)

When there is no evidence

or “Now where did I leave my glasses?”

An engineer, a mathematician and a physicist look out of a window of a train passing through the highlands of Scotland, and see a black sheep.

“Ah!”, says the engineer, “Look, they have sheep in Scotland!”.

The physicist looks at it and reflects “Well, we can say only there is at least one black sheep in Scotland”.

The mathematician looks at them both in surprise “That’s not right at all! All we can tell is that one side of one sheep in Scotland is black”

Ho. Ho.

This is one in a series of posts , about evidence and how it does or does not support a claim.

Such pedantry is over the top, but it serves to illustrate a simple point about what we can really infer from a piece of evidence, and its limitations.

It’s easy to decide what to believe when you have clear positive evidence in your hand: the photo of your girlfriend in bed with that sysadmin from finance is fairly firm (heh) evidence of her faithlessness.

It’s not so clear when there is no evidence for something, and interestingly there is more than one way in which we can not have evidence for something.

‘Bounding’ what we don’t know

Those who are old enough will know what it’s like: we lose our reading glasses, or the screwdriver, or pen or mug of tea we had in hand only a few minutes ago.

At this stage, if somebody rather stupidly asks “Well where did you leave them?” we would rightly and angrily reply “I don’t know, if I knew where they were, I wouldn’t have lost them”.

So we don’t know where they are, but we do know some places where they are not. They are not on Mars. And although we may not remember all the rooms we’ve been to since we last remember having them, we tend to remember unusual places; mine are not, for example, in the attic as I know I haven’t been there.

This gives us some limits, some ‘boundaries’, to the area of ignorance.

Reducing the area of ignorance

As we start to look we start to limit these boundaries further.

A quick walk around the usual rooms glancing at the surfaces for example is a good first stage search; it covers a lot of ground for a fairly likely result.

By discounting the places we’ve looked – and by starting with the most likely and easily surveyed places – we reduce the places the glasses could be.

“Knowing it is not” is not “Not Knowing”

Having thoroughly searched the mostly empty fridge, I know to a high degree of confidence that my glasses are not in there.

I have no evidence of that, and I have no proof that they are not (I may have forgotten to search the bowl of three week old leftover gravy) but my memories of looking are ‘evidence of no glasses in the fridge’ (‘evidence of lack’) rather than ‘no evidence of glasses in the fridge’ (‘lack of evidence’).

The latter though is still how you might reply to “Are they in the fridge?”, even though it doesn’t capture whether you’ve looked or not.

This causes problems when people want to know if there’s any danger in some treatment or chemical. To be told “There is no evidence of any harm” is useless; it doesn’t tell us if nobody’s checked, or if they’ve had a quick look and everything seems fine, or they’ve had a really very thorough search that would have turned up any significant harm and found nothing.

What we think we don’t know

So I continue with my exercise in limiting my ignorance, hoping one day to find my glasses so that I can carry on doing what I was doing… whatever that was… it will come to me in a moment… and sometimes we get a bit irrational. How many times, frustrated, have we looked in the same box, under the same small piece of paper that couldn’t possibly hide a pair of glasses?

Similarly our boundary reducing exercise is not ‘certain’; it may be I’ve looked somewhere but not seen them (after all, I’m not wearing my glasses). It may be I’ve looked in an area where they are hidden, and have declared and marked the whole area ‘glasses free’ when in fact it is not.

In more general terms, not finding doesn’t necessarily mean it’s not there: just because all the swans I’ve seen are white, does not mean there are no black swans.

This is where we reach the limits of our understanding of our limits of our ignorance. We rarely properly match the boundaries of what we think we don’t know with what we actually don’t know.

This gap is where my lost glasses still lurk when I give up and use an old pair: in the world of places I haven’t looked well enough, but can’t think of to look.

Theories of what might be

I was slightly too certain above about where my glasses are not, as aliens might have stolen them and taken them to the Mars.

And if we return to the railway carriage with the sheep-observing pedants, we might claim that “There are luminous pink sheep in Scotland with legs on one side shorter than the other. Scostmen hunt them down and turn them into haggis and bagpipes”.

We can make up any silly story we like (“Aliens live in clouds!” “Pixies ate my hamster!” “Magnets healed my cancer!” “Hair loss makes you sexy!”) and some may be accidentally true but it’s no more sensible to assume they are true without evidence than it is to believe in Garibaldi Mountain Shrews.

Not knowing something is no excuse to make up any old thing and then believe it to be true, any more than it is to believe that your glasses are in the kitchen, because you don’t know where they are, and they could be.

This is the ‘out’ for a lot of so-called ‘open minded’ views: “Just because you haven’t seen pixies, you must be close-minded to disbelieve them”.

Pixies might exist, this is true.

But when you consider all the things that might exist, such as invisible baby-eating multi-coloured pixie-swans that live with aliens in clouds, then you can see that believing in any random made up fantasy can be fun but it’s not very practical.

If someone tells you some far fetched story and says “well, you’ve got no evidence against it, so it could be true couldn’t it?” then the answer is “yes, and aliens are painting your ears”

“I don’t know”

A straightforward ‘we don’t know’ seems a bit of a cop out, and the mind abhors a vacuum, but this is no excuse to fill it with speculation and then infer ‘truths’ from them. (It’s fine to speculate and test: Perhaps the glasses are in the bathroom? I shall go and look)

Even if you’ve only ever seen white swans, you can’t be sure that all swans are white. You just might not have seen one that isn’t.

Yet lack of evidence is not evidence of lack; just because we haven’t seen something doesn’t mean it’s not there.

It’s alright to say we don’t know. It’s alright to say we think things are likely, or unlikely, but we’re not sure. And working out what we don’t know – or what exactly we’re not sure about – tells us a very valuable thing: what we still need to find out in order to know.

Background evidence

The above of course is a bit “simple”. It ignores all the background evidence we hold; there are very few sheep with black on one side and white on the other, so we can happily infer that a sheep is black from seeing the one side of it that is. This is material for another post…

The Equality Mistrust

Inequality within a society, we are sold, results in all sorts of Bad Things such as more and greater violence, poorer health, and more alcohol problems.

At first sight this would seem a bit odd. Marginal, subsistance societies where everyone is equally and awfully poor have infamously low life expectancy and high infant mortality rates. As a society’s wealth increases its health increases, and we can see this crudely with GapMinder; both over time and over space, richer societies show no obvious general trend to be any less ‘equal’ than poorer ones (in fact possibly maybe a bit the opposite), and do show an obvious trend to be healthier:

The Equality Trust

Still, the Equality Trust (“Because more equal societies work better for everyone”) are trying to make a case that ‘at some point’ – which just happens to be now in the UK and other western worlds – we can stop working on progress and instead focus on being fair. Because there are some indications that rich societies that are less ‘equal’ are also less healthy, more violent, etc:

And if you click on that you can look at various other handy graphs.

Lots of skeptical alarm bells ring: there are no units on some of the graph axis, so what are the scales? There are few dates. Countries appear on some graphs and not others. And there’s the general background that “unfairness is bad, mkay?“

In fact all that these graphs tell us is that some countries are quite consistent. The USA tops out as the least equal and most nasty, Singapore as the most equal and most nice, Japan as both equal and nice. It seems quite probable that there is something else at work here, such as differences in cultural approaches to wealth and health and social responsibility and so on.

Still, they Trust claim that “Compelling new evidence shows that large income inequalities within societies damage the social fabric and quality of life for everyone”. Presumably the well-established evidence that more income improves the quality of life is now old and optional.

Physical health vs inequality

So let’s have a look at a sample of their evidence. The paper on the first page in their evidence page, by Wilkinson and Pickett, Income inequality and population health: a review and explanation of the evidence has a title that tugs at those barely stilled skeptical alarm bells again: they are setting out to show something, not investigate. A reason to be skeptical, but not to disregard.

The abstract simply counts papers that they’ve found. There’s no indication of any methods to compensate for positive publication bias, or indeed any ordinary collection bias. They just went looking for papers, and found 155.

They then count papers and bin them according to ’supporting’, ‘unsupporting’ and ‘partially supporting’. There’s no indication of scaling or weighting factors. So a paper that looks at one factor in a specific environment is either ’supportive’ or ‘unsupportive’. A paper that considers more complex relationships is likely to find a bit of both, and so will be ‘partially supportive’. If you throw those inconvenient complicated relationships away, and look at just the simple positive findings vs the negative findings, then it should be completely unsurprising that 70% are positive, simply due to publication bias (though a ’supportive’ positive finding is not necessarily the same as a positive result that biases publications).

It’s not at all right, on the basis of that review, to conclude that the literature shows even a link (let alone a causal link) between inequality and Bad Things. In the paper they go a bit deeper than that, but not by much. The studies included were from three previous reviews, electronic searches and informal contacts, thereby introducing various collection bias.

Self-enforced relationships

There’s an interesting circular argument on page 4: Wilkinson (in another paper) points out that communities across larger areas show more correlation between poor health and inequality. He argues that this is because people can see further across differences in that community, and so see more inequality, and we should measure across wide areas to capture this. By then showing that studies of communities across larger areas return such correlation, he thinks he bears out his argument that was formed as a possible explanation for this observation…

Smilarly at the bottom is an even more circuitous argument: Inequality relates to health as a measure of the social distances which are responsible for class differences in health (ie, the social distance here is measured in healthiness). Lo and behold, if you measure those social inequalities in that particular way, you will get differences in health. Lo further and behold more, the greater those social inequalities are measured to be, the greater the differences in health. Which is what you defined it as being in the first place.

On Page 7 they assume that income ‘inequality’ is the reason why Mr Average Black in the US has four times the income of Mr Costa Rican but a lifespan shorter by nine years, and ignore a host of other social factors.

Finally – at least as finally as I got to – was that “the relationship between income inequality and homicide shows beyond doubt that inequality has powerful psychosocial and behavioural effects” (my emphasis). This is not only very poor science (correlation does not show anything beyond doubt, certainly not cause), but is also wrong: this paper (“Income Inequality, Poverty, and Homicide across Nations “) says homicide depends on absolute wealth, not equality. Although this one (“Income inequality and homicide rates in Canada and the United States“) says otherwise.

Some speculation & Some other evidence

We should in fact expect to find “fairness” as some kind of indicator, or proxy, for health. When comparing countries with similar average income, a greater spread probably means the poor are poorer and the rich are richer. The poorer then will have more of the illnesses associated with poverty, while the richer benefit from much smaller improvements in health, and so countries with a greater spread in income will have worse overall health compared to others with the same overall wealth.

All we are seeing is the diminishing returns of increasing wealth on increasing health. It’s not the inequality that causes health and crime, it’s that less equality probably shows the presence of poorer people, who have worse health and often more crime.

A proper way to measure whether societies are ‘doing the right thing’ is to compare like with like. How are those who earn $1000/month “buying power” in the US comparing with those who earn $1000/month “buying power” in Nigeria, or Singapore, or the UK? As a society gets richer – and supposedly less equal – how do the people in the bottom quartile do as it does so?

If we take a look at gapminder, and compare the Gini ‘inequality index’ with life expectancy or infant mortality (do press Play to avoid being diverted by particular years), then you can see that there’s very little correlation in general between inequality and poor health:

Similarly compare the “Income share by the lowest 20%” with life expectancy and there’s little correlation.

Only very specific years show the correlations the Equality (Mis)Trust claim, and are quickly swamped by other effects as time goes on. In fact, we can pick the year 2000 and show that richer societies are more equal:

What is more clear is the way that different parts of the world tend to sit in similar areas of the graphs. Cultural attitudes again perhaps.

What were they thinking?!

So why does this poor analysis turn up as ‘evidence’? Well the language seems a bit of a give away:

“Because more equal societies work better for everyone”

“Compelling new evidence shows that large income inequalities within societies damage the social fabric and quality of life for everyone”

“Great inequality is the scourge of modern societies”

This is the “It’s not fair” brigade dabbling in a bit of pretend science to bolster their dogma. Anyone turning up with that old mantra and a clutch of citations to prove it is going to have a lorryload of skepticism dumped on them.

Their case avoids comparisons with poor countries by assuming that ‘now we are rich and healthy’ we can take stock and think about doing things differently. But if we compare only the rich countries, the relationship is not as obvious or as large as they claim, which raises questions again about the scales of the axis used:

Even so, only most of us are whole, rich and healthy, and it’s not clear quite how rich we could be. In 50 years time, say, with free personal fusion power, no physical disabilities, no cancer, a healthy lifespan of 120 years and personal jetpacks (at last!) we might look back with pity at the relative poverty and disease-ridden state we are in now.

All the same we have some features that absolutely poor states don’t, including a relatively carefree life. We don’t start with the spectre of externally-caused starvation and death hanging over our shoulders, which tends to produce a work ethic dedicated to improving life, a sort of ‘eye on the horizon’ and a drive to work to get richer and safer.

Instead we can support whole communities of ‘idle’; that is, we can afford to pay people to do nothing, and it’s infamously depressing and frustrating to be part of those communities, and famously hard to get out of them.

It may be that we should look at the outliers in those graphs and examine potential causes; Scandanavians for example attempt to ensure full employment rather than compensate for unemployment via the dole, and that may be how they assure better equality and better health.

Wealth makes health

But in the meantime, the idea that greater inequality may sometimes be a proxy or indicator for the presence of lower absolute wealth in a society is trivial, and should not divert us from working to make everyone richer.

In particular, we must be wary of any argument that to reduce Bad Things or increase happiness, we could reduce the inequality by, say, limiting the income of the richer part of society, rather than by increasing the wealth of the poorer. As we can see above, health improves with wealth. The only happiness that such policy improves are of those who have no interest or ability to become rich, at the expense of all those who have, including the poor who are working to become rich.

The evidence remains, as can be seen above, that people living in an unfair rich society are much better off than those living in a fair poor one.

Humans Don’t Do Science

“Science”, we are frequently and rightly told, relies on being open about what it is doing and how, on welcoming informed criticism, on being willing to drop discredited ideas, on experimenting to test theories to try and break them, and generally progresses by proving existing ideas are not (quite) correct.

(This assumes a modern somewhat subverted meaning of ’science’ which will purple the pedants, but it will do for now. As will this:)

‘Scientists’ are those who do the research that brings us more and better science.

The implication then, is that as science needs the above to work, and scientists do science, therefore scientists are open, willing to drop their concepts when discredited, welcome informed criticism, and so on.

Wot tosh.

Humans eh?

Scientists are human, and so are as selfish, greedy, proud, sociable, sensitive, prejudiced, noble, dislikable, charming, arrogant when given half a chance, and generally as emotionally involved as other humans. And some scientists seem to be unaware that this breaks the requirements to ‘do science well’.

It should surprise nobody that people get emotionally involved in their work, especially if it requires a lot of effort, some specialised skills, and the results look good and are valued. This applies to most of us, and it applies just as well to a scientist who has developed a respected theory. Nobody welcomes criticism of work they are proud of.

Reputations are based on theories and ideas too, not directly on rigour in the workplace. Newton is remembered for his observations of motion (and a mythical accident for discovering them), not because of his work practices.

Similarly sometimes a huuuge amount of time, effort and money and reputation is invested in developing certain concepts, and few people can be objective when assessing their own life’s work. Skills and knowledge are accumulated and not willingly abandoned. Dark matter, neutrinos, the search for the Higgs Boson, for example, are all current research programmes that might turn out to be a complete waste of time, but there is a tremendous momentum in pursuing those particular concepts.

And so sometimes there are quite large communities of people that are emotionally invested in certain concepts. Since funds are often limited, these communities can be quite large proportions of the overall field (The CERN experiments suck up quite a lot of the physics community’s funds). If we ask certain slices of the research community what the ‘consensus’ is on a topic, the results are biased by the various investments of these communities.

Iterative Steps

Of course the key here is that we are looking at research, where we are investigating things we don’t know very well. As soon as we run a proper experiment to test the theory, then the people-yness of those involved becomes nearly irrelevent.

In the meantime we can perhaps rely on the ‘iterative’ nature of science; that we can count on the overall continual reviews to eventually correct mistakes and improve on theories. This, though, is not a set of incremental improvements, where we gradually work our way closer to the ‘truth’ in the manner of many mathematical iterations. Some models have to be completely abandoned, not just improved on.

Such a messy approach is perhaps fine for general research, but is insufficient if we need to act upon it. In some cases (such as education, climate change, materials to build bridges, buildings and airplanes) we need to assess what we actually know, and know now, from amongst all the people-y assumptions and reputations and opinions.

Being Scientific

One of the key aspects of really scientific disciplines, including ones outwith research, is that we remove the people-yness of those involved as much as possible.

For example, we record all the data and methods (“audit”) because we expect to make mistakes, and so we need to be able to go back and check every step.

We let others have access to this (“full disclosure“) because, again, we expect to make mistakes, and so we need to let other people check every step. It also helps to compensate for some of the ordinary people problems; if you know the details of your work are going to be scrutinised by all and sundry, you tend to be much more careful with that work, and much more careful with drawing conclusions from it.

We run formal assessment reviews to check methods, data sources, and citations.

Where possible, experiments are designed to remove ordinary personal biases, such as the ‘double blind trials’ used to test whether medical treatments work.

These extra tasks are tracked and checked and recorded, to make sure they are done.

Except that we don’t even do all these very well. It’s expensive, and it diverts effort from the task (even if it improves the quality of knowledge overall), and so we tend to bypass them when we can. It only tends to be properly implemented where we need very very high levels of confidence, such as medicine and bridges, buildings and airplanes, and are willing to pay for it.

It’s an odd leftover from the past that we don’t require the same rigour for informing public policy, such as in education, re-employment, and major environmental impacts.

And when scientists from some of the more ‘careless’ disciplines hold forth, we ought to consider carefully whether their views have been as openly, rigorously and systematically checked as they imply – or even believe themselves.

(“The Golem: What you should know about science” is a much more thorough take on the above)