So over at ars they had an article clearing the scientists at the CRU (the guys with the leaked emails) saying that while other statistical methods and approaches would have been superior that’s hardly misconduct. In the conversation about after the post someone was complaining about the ‘absurdity’ of not requiring the highest quality analysis and the most bulletproof possible experimental evidence for an issue like climate change. I was immediately struck by the huge gulf, which on reflection I suspect is widespread, between how people treat scientific evidence and claims and how they treat everyday questions of truth and falsity in their own lives.

However, science is nothing but the process of deciding what you think is true and if people really understood science in this way I think we’d have a lot less confusion about global warming. There is nothing different about the way disputes in science are settled than the way you might settle an argument with your friend about whose route from work to the bar is faster (suppose you have serious money riding on the question so it really matters). So let’s just consider what you’d likely do to decide the bet and see how that would apply in the scientific context. I’m using climate change here as a particular example but virtually everyone who isn’t close to the scientific community is equally guilty as this disconnect underlies many of the unreasonable public expectations about science.

When the argument begins you’d both point out factors that give credence to your route being the faster one based on existing knowledge. Often that alone will be enough to settle the question. If your friend reminds you about the new stoplights the city added on your route recently you might well decide he’s almost surely right and it’s not worth putting to the test. Or maybe his arguments are strong enough that you’d demand very favorable odds to make another bet on your route but you are still unsure enough you think it’s worth testing. (Or you could have the better argument or neither of course).

The next thing you would do do settle the question if debate persisted would be to perform an experiment. You’d probably suggest that you each drive your own routes to the bar that night and see who gets their first. Of course this kind of rough and ready check is far from an ideal controlled study with powerful statistical analysis but whether or not you the experiment convinces you depends on not only it’s quality but also the strength of the results and your prior judgments of plausibility.

In other words if your friend’s remark about the stop lights made you think he was probably right and then he reached the bar in 15 minutes while you took over 30 you’d probably find that quite persuassive and take the issue to be settled. If the results were closer, say 23 vs. 30 minutes and you know your friends a calm steady driver you might still be convinced despite the lack of experimental control for driver differences but if you know he’s highly competitive and given to speeding you might insist on having a mutual friend drive both routes to control for driver speeding. Still, as long as the results aren’t in conflict with your prior expectations, e.g., it’s the shorter, fewer stops route that is coming in as faster you’d probably be convinced without any need to throw powerful statistical methods at the question even though losing cost you a big chunk of change.

On the other hand the very same experimental results probably wouldn’t be enough to convince you if they were in radical conflict with your prior knowledge about the workings of traffic and driving. For instance if google maps said your route was shorter, had a higher speed limit, fewer stops and less traffic you’d probably suspect that the couple of tests you’d done weren’t convincing enough and demand many trials with a test of statistical significance (or compelling theoretical justification of why your route might be slower…say being told all your turns are unprotected left turns while he has right turns) before you agreed to pay up on the bet. The lower the prior probability you assign to the observed outcome and the closer the observed average times are the more statistical rigor you would insist on before paying up.

Importantly, while better controlled experiments with more powerful statistical techniques (if understood) are always better it doesn’t mean other tests can’t add to the weight of evidence. Even if you and your buddy got a third party to do many trials and used tests of statistical significance if your buddy Bob tells you he drives your way and always gets their after his wife who drives your friends way it still adds to the strength of your case despite the low number of trials and uncontrolled variables like the driver (just much less so).

So let’s take this back to the climate science situation. Would it be even better if the CRU scientists had used more powerful statistical techniques correctly? Sure. But all experiments are imperfect but they can still provide valuable evidence just like the initial tests or Bob’s anecdote in my story. Not only is it not bad for scientists to publish results from imperfect experiments it’s important for them to do so as long as other scientists aren’t being mislead about the strength of the tests/results (the scientific community seems unsurprised about the underlying statistical techniques used so it’s reasonable to assume they probably had a good sense of the strength of the evidence published). In fact often it’s preferable for scientists to publish imperfect experimental results with non-ideal methods of analysis and invest their time in new experiments instead of investing the time/energy to apply more powerful mathematical techniques to a particular result. For instance I think it was probably a good decision on Galileo’s part to let the crude experiments with balls on inclined planes and simple timers convince him objects all accelerate at the same rate under gravity than to insist on waiting for fancy statistics and running many tests. That way he could say important things about the heliocentric model as well.

Ultimately what’s wrong with the expectation that papers giving evidence on matters of enormous public concern always live up to some kind of school book ideal of scientific experimentation is that it makes a single paper stand in for the mountains of accumulated evidence. I mean complaining about all these minor ‘imperfections’ in climate research is like arguing that we shouldn’t assume it takes longer to drive from New York to LA than from Chicago to LA because the people who had made the drives never had a proper control. If you have shitloads of distinct pieces of evidence for a claim (like many people saying 2 days for Chicago LA and 3 days for NY LA) which is supported by a sound theoretical justification (NY is farther away) demanding the tests be redone with the right formal methods (controls/better statistics/etc..) would make no noticeable difference in your (rational) confidence of the claim.

Equally problematic is the background assumption that most public criticisms of consensus scientific results tend to ignore the critical importance of prior probabilities. For instance if there was one thing I think that most people on both sides of the climate debate ignore (or glass over) is the high prior probability we should assign to anthropogenic global warming. Basic thermodynamic considerations tell us that absent some unexpected large systematic effect of CO2 emissions to create cooling it must cause global warming and that human emissions are large enough that we should expect them to give rise to the level of warming observed. This isn’t some crazy computer model or complex bit of climate science subject to judgment calls. Just look up a few basic values for light absorption by CO2, strength of solar radiation and the (directly observed by satellite) level of reflected solar radiation and you can work it out on a single piece of paper.

So it’s the guys who want to deny anthropogenic global warming who have the burden of proof to convince us there is some unexpected process that gives rise to a seemingly paradoxical absence of anthropogenic global warming. Given this context merely suggestive experimental results supporting the accuracy of this simple model are enough to up our confidence from “likely” to “almost certain.” These scientists should no more waste time polishing these experiments than they should spend money rigorously verifying the earth is round. In other words we already have SOOO much evidence that making this paper a bit more persuasive wouldn’t budge our rational confidence.

A similar point applies to resistance to things like food irradiation or other all natural type movements. As with climate change the prior probabilities are ignored and the burden is shifted from the person suggesting the possibility of novel unexpected mechanisms to those who are simply claiming that things work the way we reasonably expect them to work.