For the second time in a week I have removed the word “significant” from a draft manuscript written by a colleague of mine in clinical medicine. In a significantly p’d I wrote about the myth of significance – that is about the ubiquitous use of the term “significant” in the medical literature to mean a specific probability incorrectly rejecting the hypothesis that two things (eg two treatments) are the same (you may need to read that twice). What I pointed out was the “significant” does not mean “meaningful.” Here I want to propose an alternative. But first, I need to discuss two major problems with the term.
Where common is not specific
In my experience the common usage of “significant” to mean important is the normal interpretation of the word in the science literature even by many medically trained people and sometimes the authors of articles themselves.
The tyranny of p<0.05
When the maths wiz Ronald Fisher talked about significance (in an agricultural journal not a medical one!) he used 0ne in 20 (p<0.05) as an acceptable error rate in agricultural field trials so that trials did not have to be repeated many times. That p<0.05 has taken on almost magical proportions (‘scuse the pun) in the medical literature is scary and shameful. I don’t want to delve into all that now. If you want to, a starting point maybe the Nature article here.
I propose that in all scientific literature that authors replace the term “significant” with the phrase “beyond reasonable doubt” and that they only be allowed to publish the article if in the methods section they define what p value they choose to represent “beyond reasonable doubt” and they defend why they have chosen this value and not another. “Beyond reasonable doubt” is a term used in the New Zealand judicial system where those charged with a crime are presumed innocent (Null hypothesis) until proven otherwise. Perhaps those of us in science could learn something from our lawyer friends.
I expect New Zealand tax dollars science spending to be better than what I read in the media this morning. Headlined in the Press was “Blenheim ‘fat mate’ loses 13.5kg in 8 weeks.” The story was of someone on a trial of a locally produced diet supplement having lost weight. So far nothing to peak my interest, but then I came across the statements “Satisfax was the result of $12 million research over four years with support from Crown Research Institute Plant & Food Research.” (Satisfax is the trademark) and “Huge demand for the trial saw it expanded from 100 “fat mates” to 200.” The second of the links goes to an October article in the Marlborough express which includes the statement that “The trial had been approved by the Health Ministry’s health and disability ethics committee and was partially funded by Callaghan Innovation.”
So, your and my taxes are being spent by Callaghan Innovation on a trial of a diet pill the development of which received other tax dollars through Plant & Food. Worth a little more investigation. The trial went through an ethics committee – big tick. It was also (a little late) registered on the Australia New Zealand Clinical Trials network (here) – tick.
BUT, it fails miserably as an efficacy trial.
There is no control group. ie the pill is not compared against a placebo. I can think of no practical reason why there was not a control group taking a placebo (randomised and blinded of course). Instead, the trial just looks at the average change in weight change over eight weeks and tries to establish if this is non-zero. Given that these people are doing something hoping to loose weight, there may well be an average loss of weight that has nothing to do with the pill. The press article suggests a biostatistician is going to somehow “account” for the placebo affect (something not mentioned on the trial registration). I pity the biostatistician as this is involves trying to convince someone that a study run elsewhere with a placebo group, at a different time, under different circumstances could actually serve as a control for this study.
Incredibly, that is not the only major issue. I read that part way through the trial the publicity was such that there was a demand from people to enter the trial and so they doubled the number of participants from 100 to 200. Ahhhhhh….. this is classic introduction of bias and should never have been allowed. Those extra 100 people are not the same as the first group… they have elevated expectations that may well bias the results. Furthermore, it is always dangerous to talk about the trial efficacy part way through as this may influence the behaviour of those already in the trial. Grrrrr….
In short – a chance lost and waste of Plant and Food & Callaghan Innovation funding. There is hope though – a proper randomised controlled trial could be conducted. But, I won’t be holding my breath.
ps. The Marlborough Express and Press should be ashamed of such blatant product placement – diet pills on January 2nd are so cliche. I wonder if it was the reporter or the company who initiated this piece?