Sound policies rely on sound data, but statistics can be very misleading.  I strongly recommend that all policy makers should read David Spiegelhalter's The Art of Statistics:  Learning from Data - a wonderfully accessible text for non-statisticians,  Here are summaries of some of the problems that he discusses.

So what is to be done? If possible, you should design your own questionnaires and data-gathering exercises with the help of professional statisticians. To the extent that this is not possible, you must treat all data with a heavy dose of cynicism, bearing in mind all the issues listed above.

But do not be tempted, when faced with a hostile press or a one-sided lobby, to assemble your own dodgy statistics – or dodgy science – to fight them off. The inevitable result would be that those with whom you are trying to communicate would then see you as prejudiced and/or adversarial, and you might also then fail to pay insufficient attention to perfectly reasonable arguments from ‘the other side’.  Instead, follow this link to read and apply some excellent advice about how to communicate statistics.

Nutrition Research

Several of the above problems bedevil scientific 'advice' about diet.  New Scientist magazine's Clare Wilson published an interesting article on this area in July 2019.  Here is an extract:

The big problem with these “observational” studies is that eating certain foods tends to go hand in hand with other behaviours that affect health. People who eat what is generally seen as an unhealthy diet – with more fast food, for instance – tend to have lower incomes and unhealthy lifestyles in other ways, such as smoking and taking less exercise. Conversely, eating supposed health foods correlates with higher incomes, with all the benefits they bring. These other behaviours are known as confounders, because in observational studies they can lead us astray. For example, even if blueberries don’t affect heart attack rates, those who eat more of them will have fewer heart attacks, simply because eating blueberries is a badge of middle-class prosperity.  Researchers use statistical techniques to try to remove the distorting effects of confounders. But no one knows for certain which confounders to include, and picking different ones can change results.

To show just how conclusions can vary based on choice of confounders, Chirag Patel at Harvard Medical School examined the effects of taking a vitamin E supplement. He used a massive data set from a respected US study called the National Health and Nutrition Examination Survey. Depending on which mix of 13 possible confounders are used, taking this vitamin can apparently either reduce death rates, have no effect at all or even raise deaths. Patel says this shows researchers can get any result they want out of their data, by plugging into their analysis tools whatever confounders give an outcome that fits their favoured diet, be it low-fat or low-carbohydrate, vegetarian or Mediterranean.. ...

Another source of error is known as publication bias:   studies that show interesting results are more likely to get published than those that don’t. So if two studies look at red meat and cancer, for instance, and only one shows a link, that one is more likely to be published.  This bias happens at nearly every stage of the long process from the initial research to publication in a scientific journal and ultimately to news stories, if journalists like me write about it. “What you see published in the nightly news is the end result of a system where everyone is incentivised to come up with a positive result,”.

Further Reading

There are lots of good books about the use and abuse of statistics, written for non-experts.  As well as almost everything by David Spiegelhalter, I can safely recommend almost everything by Tim Harford, including How to Make the World Add Up.

Martin Stanley

Spotted something wrong?
Please do drop me an email if you spot anything that is out-of-date, or any other errors, typos or faulty links.