beyond reasonable doubt: a significant improvement

For the second time in a week I have removed the word “significant” from a draft manuscript written by a colleague of mine in clinical medicine. In a significantly p’d I wrote about the myth of significance – that is about the ubiquitous use of the term “significant” in the medical literature to mean a specific probability  incorrectly rejecting the hypothesis that two things (eg two treatments) are the same (you may need to read that twice).  What I pointed out was the “significant” does not mean “meaningful.”   Here I want to propose an alternative.  But first, I need to discuss two major problems with the term.

Where common is not specific

In my experience the common usage of “significant” to mean important is the normal interpretation of the word in the science literature even by many medically trained people and sometimes the authors of articles themselves.

The tyranny of p<0.05

When the maths wiz Ronald Fisher talked about significance (in an agricultural journal not a medical one!) he used 0ne in 20 (p<0.05) as an acceptable error rate in agricultural field trials so that trials did not have to be repeated many times.  That p<0.05 has taken on almost magical proportions (‘scuse the pun) in the medical literature is scary and shameful.  I don’t want to delve into all that now.  If you want to, a starting point maybe the Nature article here.

My proposal

I propose that in all scientific literature that authors replace the term “significant” with the phrase “beyond reasonable doubt” and that they only be allowed to publish the article if in the methods section they define what p value they choose to represent “beyond reasonable doubt” and they defend why they have chosen this value and not another.  “Beyond reasonable doubt” is a term used in the New Zealand judicial system where those charged with a crime are presumed innocent (Null hypothesis) until proven otherwise.  Perhaps those of us in science could learn something from our lawyer friends.

Fat Mate: Fat chance lost

I expect New Zealand tax dollars science spending to be better than what I read in the media this morning. Headlined in the Press was “Blenheim ‘fat mate’ loses 13.5kg in 8 weeks.”  The story was of someone on a trial of a locally produced diet supplement having lost weight.  So far nothing to peak my interest, but then I came across the statements “Satisfax was the result of $12 million research over four years with support from Crown Research Institute Plant & Food Research.” (Satisfax is the trademark)  and “Huge demand for the trial saw it expanded from 100 “fat mates” to 200.”  The second of the links goes to an October article in the Marlborough express which includes the statement that “The trial had been approved by the Health Ministry’s health and disability ethics committee and was partially funded by Callaghan Innovation.

So, your and my taxes are being spent by Callaghan Innovation on a trial of a diet pill the development of which received other tax dollars through Plant & Food. Worth a little more investigation.  The trial went through an ethics committee – big tick.  It was also (a little late) registered on the Australia New Zealand Clinical Trials network (here) – tick.

BUT, it fails miserably as an efficacy trial.

There is no control group.  ie the pill is not compared against a placebo. I can think of no practical reason why there was not a control group taking a placebo (randomised and blinded of course).  Instead, the trial just looks at the average change in weight change over eight weeks and tries to establish if this is non-zero.  Given that these people are doing something hoping to loose weight, there may well be an average loss of weight that has nothing to do with the pill. The press article suggests a biostatistician is going to somehow “account” for the placebo affect (something not mentioned on the trial registration).  I pity the biostatistician as this is involves trying to convince someone that a study run elsewhere with a placebo group, at a different time, under different circumstances could actually serve as a control for this study.

Incredibly, that is not the only major issue.  I read that part way through the trial the publicity was such that there was a demand from people to enter the trial and so they doubled the number of participants from 100 to 200.  Ahhhhhh….. this is classic introduction of bias and should never have been allowed.  Those extra 100 people are not the same as the first group… they have elevated expectations that may well bias the results.  Furthermore, it is always dangerous to talk about the trial efficacy part way through as this may influence the behaviour of those already in the trial.   Grrrrr….

In short – a chance lost and waste of Plant and Food & Callaghan Innovation funding.  There is hope though – a proper randomised controlled trial could be conducted.  But, I won’t be holding my breath.

ps.  The Marlborough Express and Press should be ashamed of such blatant product placement – diet pills on January 2nd are so cliche.  I wonder if it was the reporter or the company who initiated this piece?

Cheesecake files: Just how deadly is it?

Everyone said it did, but how did they know and by how much?  Statements like

“The development of AKI [Acute Kidney Injury] after CPB [Cardiopulmonary Bypass Surgery] is associated with a significant increase in infectious complications, an increase in length of hospital stay, and greater mortality.” (Kumar & Suneja, Anaesthesiology 2011 14(4):964)

are common place in the acute kidney injury literature.  When I started to look at the references for such statements I realised that they were all to individual, normally single centre, studies and that the estimates of the increased risk associated with AKI after CPB varied considerably.  Furthermore, the way AKI is defined in these studies is quite varied. This lead to two questions?

  1. Just how deadly is getting AKI after CPB?
  2. Does it matter how we define AKI in this case?

These questions are important as the answer to them helps a surgeon and patient to better assess the risk associated with choosing to have cardiopulmonary bypass surgery and what the importance is in monitoring kidney function after such a surgery.  To answer these questions required a meta-analysis the results of which I have just published (a.k.a earned a cheesecake).  A meta-analysis involves systematically searching through the literature, a sentence which takes seconds to write but months to serve, for all articles reporting an association between AKI and mortality after CPB.  Then there is learning how to put all the, sometimes disparate, data together (I had to learn a lot of R for this one) and to report on it.  As this was my first meta-analysis, I was fortunate to have the assistance of two highly competent scientists & nephrologists with meta-analysis experience, namely Dr’s Matt James of Calgary, and Suetonia Palmer of my own department in the University of Otago Christchurch.

So – what did we find?

  1. If you get AKI after CPB you about 4 time more likely to die compared to if you do not get AKI after CPB even after accounting for things like age, diabetes, and other risk factors.
  2. Somewhere between 37 and 118 lives per 10,000 CPB operations could be saved if we could find a way to eliminate AKI.
  3. How AKI was measured did not make any difference to the results.
  4. AKI after CPB was also associated with increased risk of stroke.
Figure 1 from Pickering et al, AJKD 2014

A teaser of a figure from Pickering et al, AJKD 2014

Pickering, J. W., James, M. T., & Palmer, S. C. (2014). Acute Kidney Injury and Prognosis after Cardiopulmonary Bypass: A Meta-analysis of Cohort Studies. American Journal of Kidney Diseases : the Official Journal of the National Kidney Foundation. doi:10.1053/j.ajkd.2014.09.008

ps. Sorry about the paywall folks, but as I’ve said before, if we want to put this data in front of the people it is most relevant to we haven’t the budget to always make them Open Access.

 

Can Doctors and Nurses help Dialysis patients recover?

In the case of dialysis dependent acute kidney injury patients this is a question which Dr Dinna Cruz  and colleagues (University of California San Diego) are asking and seeking opinions from both nephrologists and non-nephrologist doctors and nurses involved in care of dialysis patients.  It was a question which arose out of discussions at this year’s Continuous Renal Replacement Therapies conference (CRRT 2014). Personally, I think it is a brilliant starting point for research to go out and seek the opinion of those “at the coal face” actually treating patients. If that includes you, please take a moment to complete the survey. If it includes someone you know, please pass this request to participate on.  Here is Dr Cruz’s request:

Currently there is much interest regarding the recovery aspect of AKI. A specific area of interest is how to enhance recovery in patients who remain dialysis-dependent at the time of discharge. It is hypothesized that patients with potential for renal recovery may require a different care plan than the “usual” ESRD patient.

Therefore we are asking your opinion regarding the post-discharge care of such patients, using this short survey. It will take only a few minutes of your time, and represents a starting point for developing potential strategies for these patients. We think it is very important to have the input of specialists from different healthcare settings and countries to give a more balanced view.

Kindly complete the survey appropriate for your specialty, then please share both these links with other colleagues so we get more responses from around the world

For nephrologists:

https://www.surveymonkey.com/s/postdischAKIcare_neph

For non-nephrologists, including acute and chronic dialysis nurses:

https://www.surveymonkey.com/s/postdischAKIcare

Thank you very much for your help!

Source: Anna Frodesiak-Wikimedia Commons

Source: Anna Frodesiak-Wikimedia Commons

Publication police and how to choose where to publish

“I confess, I published behind a paywall.  I’m sorry, sir, I didn’t want to, but but but I’m almost out of funds and and …..<suspect’s voice fades>”             Publication Police files, Nov.3 2024

Will peer pressure eventually lead to discrimination against those who publish behind a paywall?  Is it now a moral imperative that we publish everything open access?  If so, is that not simply morality by majority (a dangerous proposition at the best of times), or worse, morality by the most vocal?

I’m often asked “Where should I publish this?” and I must admit that “In an open access journal” is not my first response.  This is simply because there is a higher standard than mere open access (as great as that is).  Where to publish is first and foremost the answer to the question “Where will it get the attention it deserves?”  Of course, this is where ego can raise its ugly head and, worse, I have colleagues who think this means the journal with the highest impact factor, but those distractions aside, it is still the most important question.

Most of our science is simply an incremental step building on what is going before.  Most of the time it is of interest to a relatively small group of fellow researchers or those whose profession is impacted on by the research.  Furthermore, it will probably be of interest only for a short period of time before someone else builds upon it. The “attention a paper deserves” is the attention that these people for whom it has most meaning give it.  For this reason, it should be published in a manner which makes it easy for these people to read about it and access it. This will probably mean one of the professional society journals and/or one of the most read journals in the field.  In the fields of Critical Care and Nephrology where I’ve published most recently this will probably mean a European or American journal which has high readership in those jurisdictions because this is where most of the research is being done.    Of course, this does not mean my manuscript will necessarily be accepted by those journals, but if I deem it has something important to say, then that is where I should send it first.

Comparatively few of those journals are open access only, but all offer an open access option.  This tends to come with a publishing fee in the range of US$1500 to US$3000.  My budget does not stretch to paying such a fee for every publication. I am forced to be pragmatic. If my manuscript is accepted into one of those more high profile journals I have to pick and choose.  The more important I think the findings the more likely I will take the open access option.  Also, if I think the message has immediate application for clinicians (i.e. not just the narrow group of researchers in my field) I am more likely to choose open access.

There is, of course, the option to publish in more general online journals (PlosOne, PeerJ, F1000 etc) and I have done so.  However, my impression at this stage it that these do not rapidly reach the inbox of most of the very very busy researchers and clinicians in the fields I publish in.  A few (like myself), may have set up automatic search strategies or use social media to follow journals in their field, and, of course, if people are conducting PubMed or the like searches they may come across those articles.  However, their lack of specialisation and reliance on someone making more effort over and above reading the specialised professional journals they have always read, mitigates somewhat their usefulness to me to “getting the message out.”  Of course, I could choose to be a “early adopter” or “pioneer” and publish in a low cost open access journal (if my fellow authors would let me) with the hope that this will change the publishing culture of paywalls and high publishing fees elsewhere.  However, it would be at the cost of less exposure of my research to those who are most interested and active in the field.  For some of what I publish I must balance my obligation to advance the field the most by maximising the chances of exposure amongst those for whom it is likely to be of immediate interest with the more philosophical desire for open access to all and sundry from now to eternity.

________________________________________________________________

sponsors_4

 

A funding model that works (for me)

I’ve been a strong critic of the bias towards project rather than people based funding in public grants for science research.  Now, I celebrate being the recipient of people based funding thanks to a combined initiative of the Emergency Care Foundation, the Canterbury Medical Research Foundation, and the Canterbury District Health Board. What has transpired is exactly what I believe the country needs much much more of, namely initiatives that get behind people and teams with broad goals and long term vision rather than narrow projects and annual funding angst.

Below is a press release.  Over the next 5 years I’ll share the ups and downs of the research we undertake.  I’ve already posted about a study I’m currently analysing the results of, which was designed to rapidly assess patients presenting to the ED with chest pain and to safely reduce unnecessary admissions to hospital.

On a more personal note I am grateful to Dr Martin Than (ED) who has championed this intiative and employed me to date this year, to Kate Russell and the CMRF board who have shown great support and helped to put this together, and to Carolyn Gullery and several others of CDHB Planning and Funding who have made possible those very important linkages and collaborations within the DHB. For the record, this is an 80%, 5 year, fellowship; I retain a 20% position with the University of Otago Christchurch, and will continue to undertake collaborative work with many research teams based there.

sponsors_3small

________________________________________________________

MEDIA RELEASE

Health Board and Private Health and Research Foundations join forces to improve outcomes in Acute Care Delivery

The Canterbury Medical Research Foundation, The Emergency Care Foundation and the Canterbury District Health Board have put their collective might behind a new Senior Research Fellowship in Acute Care based at Christchurch Hospital.

The five-year fellowship will improve the quality of Acute and Emergency Care delivery at the hospital through targeted research and close collaboration with clinicians eager to see positive research outcomes translated into real-world clinical practice and, importantly, into improved patient outcomes.

The demand on acute services is large, around 140,000 patient visits a year across the health system. The challenge is to quickly, accurately and safely assess a patient’s condition and to ensure they access the services they need and don’t unnecessarily access the services they don’t need. The first projects this Fellowship supports will better identify which patients presenting to the emergency department with chest pain need admission and intensive monitoring, and who can be safely discharged home.

The Fellowship will be undertaken by Associate Professor John Pickering. Dr Pickering’s research has been cross-disciplinary. This began with the application of physics in dermatology and plastic surgery through the use of lasers in medicine, in particular he helped develop the use of lasers to remove birthmarks. Over the past seven years through he has advanced the diagnostic methods to detect acute kidney injury and most recently, has become involved in research to discover and translate into clinical practice, diagnostic protocols in the emergency department, particularly for patients presenting with chest pain and the possibility of a heart attack.

Dr Pickering sees medical science as a team effort involving, not only doctors, nurses, and scientists, but also the patients themselves, and funders. He is a keen advocate that publically funded research be made known and understandable to a lay audience through blogs and social media. He writes a blog on the Sciblogs.co.nz web site as “Kidney-punch.”

The Canterbury Medical Research Foundation and Emergency Care Foundation are delighted to be partnering with the DHB on a project that is likely to have long term effects on the delivery of acute care in the Canterbury Health system and further afield.

“This type of directly translational research that will give us definite and measureable improvements in patient care is something we are particularly interested in. Committing to five years will allow enough time for research findings to be properly utilized in improvement in practice patterns in real life clinical situations and that is very exciting.” Says Kate Russell, Chief Executive of the Foundation

“I believe that this is an excellent collaborative project to better integrate medical research with clinical care delivery. This initiative will actively facilitate the alignment of some excellent medical research that is taking place in Christchurch with the Canterbury District Health Board’s priorities and plans for improving care for patients with suspected acute, and particularly cardiovascular, illness” said Dr Martin Than, Emergency Care Foundation

For Canterbury DHB it represents an exciting era of partnering with private trusts and foundations to make inroads into issues of quality improvement and better outcomes for Canterbury people.

“We have already made significant progress in reducing acute demand on our hospitals, with more than 28,911 people receiving treatment and care in the community in the past year. This research will provide important evidence to support future decision-making about how, where and which services are funded and provided to ensure Canterbury people receive the right care, at the right time, in the right place, by the right person.” Said Carolyn Gullery, General Manager of Planning and Funding at the DHB.

ENDS
For further information please contact Dr Martin Than on Martin.Than@cdhb.health.nz

It’s all about the math, dummy!

No one understands the electoral maths of the NZ electoral system including the electoral commission apparently. Last night I put the latest figures from the “Poll of Polls” into the electoral commission calculator and I discovered the calculator was broken! I put the figures in with United Future winning one electorate seat, but when it crunched the numbers it gave me a parliament without United Future in it. Hmmm… have I uncovered a conspiracy to keep Peter Dunne out of parliament, or is it just evidence that someone got their math wrong. Let’s hope it’s the latter and that they’ll get it right on the night.

Electoral Commission calculator results captured 17 September 2014

Electoral Commission calculator results captured 17 September 2014

In the meantime, let’s consider two concepts this election hangs on – the so called “Wasted vote” and the “Overhang.” The Wasted Vote is the proportion of votes that go to parties that do not make it into parliament by either crossing the 5% Party vote threshold OR by winning at least one electoral seat. The overhang is when a party or parties win more electoral seats than the proportion of their Party vote entitles them too. This means that the size of parliament would increase. Normally 120 and 1/120th of the party vote (0.83%) is equivalent to one member of a party. However, for example, if a party receives just 1% of the vote, but wins 2 electorate seats then this will increase the size of parliament to 121. The various permutations of polls have the current election resulting in a parliament ranging from 120 to 124 seats.

The number of seats in parliament is crucial because it means the effective number of seats a party of block of parties must win in order to form the majority to govern increases. 61 seats are needed for a 120 or 121 member parliament, 62 for a 122 or 123, and 63 for a 124 member parliament.

About the Wasted vote two ideas are important:

The Wasted vote supports the party already with the most votes the most

The Wasted vote could determine who governs!

Let’s assume that 61 seats are necessary in a 120 seat parliament. Ie a block needs 61/120th of the party vote (50.83%) to govern. Crucially this percentage, though, is NOT the percentage of the vote that block gain on the night (which is what the polls try and predict). What it is, is the “effective percentage” after the Wasted votes are taken into account. A scenario could help. Consider an election with two parties crossing the 5% threshold to get into parliament and all the rest being wasted votes. Let’s call the two parties the Big, Rich and Totally Selfish (BRATS*) party and the Really After Total State (RATS**) party. Consider this, there are 1 million voters. BRATS gets 450,000 votes on the night (45%). But, 10% (100,000) of the vote is Wasted. That means the proportion of votes the BRATS get out of the non-wasted votes is 450,000/900,000 giving an “effective percentage” of 50% which would give them 60 seats in parliament.  The RATS would have the same in this scenario. We can turn this question around the other way and ask how high a proportion of the total vote does the Wasted vote have to be for the BRATS “effective percentage” to cross the 50.83% threshold needed to govern? This will depend on the total proportion of votes the BRATS receive  (in our example 45%). The graph below illustrates this.

The percentage of wasted votes the BRATS need in order to govern based on the actual percentage of votes they receive

The percentage of wasted votes the BRATS need in order to govern based on the actual percentage of votes they receive

So, folks, if on the night your vote is in the waste basket, rest assured it will have an effect on the outcome of this election.  The only truly wasted vote is the one that is not cast!

_______________________________________

*Led by Mr I.M. Wright

** Led by Mr M.Y. Tern