Tag Archives: p value

beyond reasonable doubt: a significant improvement

For the second time in a week I have removed the word “significant” from a draft manuscript written by a colleague of mine in clinical medicine. In a significantly p’d I wrote about the myth of significance – that is about the ubiquitous use of the term “significant” in the medical literature to mean a specific probability  incorrectly rejecting the hypothesis that two things (eg two treatments) are the same (you may need to read that twice).  What I pointed out was the “significant” does not mean “meaningful.”   Here I want to propose an alternative.  But first, I need to discuss two major problems with the term.

Where common is not specific

In my experience the common usage of “significant” to mean important is the normal interpretation of the word in the science literature even by many medically trained people and sometimes the authors of articles themselves.

The tyranny of p<0.05

When the maths wiz Ronald Fisher talked about significance (in an agricultural journal not a medical one!) he used 0ne in 20 (p<0.05) as an acceptable error rate in agricultural field trials so that trials did not have to be repeated many times.  That p<0.05 has taken on almost magical proportions (‘scuse the pun) in the medical literature is scary and shameful.  I don’t want to delve into all that now.  If you want to, a starting point maybe the Nature article here.

My proposal

I propose that in all scientific literature that authors replace the term “significant” with the phrase “beyond reasonable doubt” and that they only be allowed to publish the article if in the methods section they define what p value they choose to represent “beyond reasonable doubt” and they defend why they have chosen this value and not another.  “Beyond reasonable doubt” is a term used in the New Zealand judicial system where those charged with a crime are presumed innocent (Null hypothesis) until proven otherwise.  Perhaps those of us in science could learn something from our lawyer friends.

Note to self – eat more chocolate

Apparently the pinnacle of one’s scientific career is to win a Nobel Prize.  Having not won a Nobel yet I will just need to accept by faith that it would indeed be the pinnacle of my career.  Thanks to a brilliant article in the New England Journal of Medicine this week I now have a new strategy to gain that elusive medal – eat more chocolate.  Dr Franz Messerli has nicely illustrated that the number of Nobel laureates per 10 million of population is correlated well with the chocolate consumption of the country of origin of the laureates (Messerli FH. Chocolate Consumption, Cognitive Function, and Nobel Laureates. N Engl J Med 2012;).  The correlation is strong with an r=0.79* (p<0.0001**) increasing to 0.86 with the removal of one outlier (Sweden). As the author wrote:

“..since chocolate consumption has been documented to improve cognitive function, it seems most likely that in a dose-dependent way, chocolate intake provides the abundant fertile ground needed for the sprouting of Nobel laureates. Obviously, these findings are hypothesis-generating only and will have to be tested in a prospective, randomized trial.”

On the outlier he wrote:

“The only possible outlier seems to be Sweden. Given its per capita chocolate consumption of 6.4 kg per year, we would predict that Sweden should have produced a total of about 14 Nobel laureates, yet we observe 32. Considering that in this instance the observed number exceeds the expected number by a factor of more than 2, one cannot quite escape the notion that either the Nobel Committee in Stockholm has some inherent patriotic bias when assessing the candidates for these awards or, perhaps, that the Swedes are particularly sensitive to chocolate, and even minuscule amounts greatly enhance their cognition.”

You may wonder why this was not published on 1 April .  Is this merely another example of “correlation doesn’t equal causation” and the tyranny of the p value (more on that in another post), or could there really be something in it? Have a read of the article and judge for yourself.

No good scientific report is worth its salt without a testimonial (here):

“I attribute essentially all my success to the very large amount of chocolate that I consume,” said Eric Cornell, an American physicist who shared the Nobel Prize in 2001.

“Personally I feel that milk chocolate makes you stupid,” he added. “Now dark chocolate is the way to go. It’s one thing if you want like a medicine or chemistry Nobel Prize, OK, but if you want a physics Nobel Prize it pretty much has got to be dark chocolate.”

*  if r=1 then correlation is perfect, if r=0 then there is nor correlation at all.  0.79 is impressive.

** this means that there is a .01% chance that the correlation observed was due to random chance.