Tag Archives: New Zealand

PBRF: The end is nigh

I’d like to say the end is nigh for the performance-based research fund (PBRF), full stop. A few months ago, I demonstrated how the expensive and tedious production of evidence portfolios by 7000 academic staff will do nothing to change the redistribution of research funding – the purported reason for PBRF. So, I’d like to say the end is nigh because the minister responsible (Hon. Chris Hipkins) has seen the light and pulled the plug. But, alas, it is simply that all portfolios have now been submitted and so await assessment by the peer review panels . About 250 people serve on these panels, nearly all of whom are Professors, most from New Zealand but a sprinkling from Australia and elsewhere.  They represent the gathering of some of the best minds in the country.  From my perspective it is a terrible waste  of time for them and of tax-payers’ money for the rest of us. 

In completing my portfolio I received a message concerning citation counts that “Panels are not a fan of Google scholar as they think the counts are over-inflated. You can use this but also supply cite counts from either Scopus or WoS.” Frankly, I think the panellists are far too intelligent to worry about this and I expect that they realise that while Google scholar counts are over-inflated, that Scopus (owned by Elsevier!) and WoS under-count (eg by not counting book chapters, leaving out some journals etc).  What matters, if citations have to be used at all, is that apples are compared with apples.  I’ve discussed some of these problems recently.  Before I suggest a solution that doesn’t require 250 Professors sitting in days of meetings, or 7000 academics spending days in completing evidence portfolios, I’ve produced a graphic to illustrate the problem of comparing apples with oranges.  Google scholar ranks journal according to the 5-year h-index. These can be explored according to the various categories and sub-categories Google Scholar uses (here). Visually each of the 8 major categories has different numbers of citations and so of the h-indices.  For example, Social Sciences is a small fraction of Health and Medial Sciences, but is larger than the Humanities, Literature & Arts.   Within each category there are large differences between sub-categories.  For example, in the Health & Medical Sciences category a cardiologist publishing in cardiology journals will be publishing in journals where the top 20 h-indices range from 176 to 56.   However, the Nursing academic will be publishing in journals whose top 20 h-indices range from 59 to 31.  So what is needed is a system that takes into account where the academic is publishing.

Visualisation of Google Scholar’s h-5 index Categories (large ellipses at the bottom) and sub-categories (smaller ellipses). Each sub-category ellipse represents in height and area the sum of the h-indices for 20 journals within that sub-category.

Google Scholar, which, unlike WoS and Scopus, is open and public, can be scraped by just three lines of code in R (a free and open programming language) to extract the last 6 years of published article and their citations for any academic with a profile on Google Scholar.  Thousands of NZ academics already have one.  Here’s the code which extracts my last 6 years of data:

library(scholar)
library(dplyr)
pubs<-get_publications("Ig74otYAAAAJ&hl") %>% filter(year>=2012 & year <=2017)

 

The “Ig74otYAAAAJ&hl” is simply the unique identifier for me which is found in the URL of my Google Scholar profile (https://scholar.google.co.nz/citations?hl=en&user=Ig74otYAAAAJ&hl).

I’ve also been able to scrape the list of top 20 journals and their h-index data for the 260 sub-categories from Google Scholar.  Here is what Cardiology looks like:

Google Scholar’s tops 20 journals for Cardiology as at 13 July 2018: https://scholar.google.co.nz/citations?view_op=top_venues&hl=en&vq=med_cardiology

So, how do we use all this data to compare academics without them having to submit screeds of data themselves?  All that needs is for them to be registered with their Google Scholar identity and for there to be an appropriate formula for comparing academics.  Such a formula is likely to have several components:

  1. Points for ranking within a category. For example, 20 pts for a publication ranked first in a subcategory, down to 1 pt for a publication ranked 20th and, say, 0.5 pts for ones not ranked.
  2. Points that reflect the number of citations a paper has received relative to the h-index for that journal and with a factor that accounts for the age of the paper (because papers published earlier are likely to be cited more).  For example, #citations/Journals 5y h-index * 2/age[y] * 20.  I use 20 just to make it have some similar value to that of the ranking in point 1 above.
  3. Points that reflect the author’s contribution.  Perhaps 20 for first author, 16 second, 12, 8, and 4 for the rest + a bonus 4 for being Senior author at the end.

Here’s a couple examples of mine from the last 6 years:

Pickering JW, Endre ZH. New Metrics for Assessing Diagnostic Potential of Candidate Biomarkers. Clinical Journal Americac Society Nephrology (CJASN) 2012;7:1355–64. Citations 101.

The appropriate sub-category is “Urology & Nephrology” (though I wonder why these are grouped together, I’ve published in many Nephrology, but never a Urology journal).

  1. Ranking:  12 points.    [CJASN is ranked 8th, so 20-8 = 12]
  2. Citations:  10.8 points. [ CJASN 5y h-index is 62. Paper is 6 years old. 101/62 * 2/6 * 20 =10.8]
  3. Author: 20 points [ 1st author]
  4. TOTAL: 42.8

Similarly for:

Flaws D, Than MP, Scheuermeyer FX, … Pickering JW, Cullen, L. External validation of the emergency department assessment of chest pain score accelerated diagnostic pathway (EDACS-ADP). Emerg Med J (EMJ) 2016;33(9):618–25. Citations 10.

The appropriate sub-category is “Emergency Medicine”  (though I wonder why these are grouped together, I’ve published in many Nephrology, but never a Urology journal).

  1. Ranking:  12 points.    [EMJ is ranked 8th, so 20-8 = 12]
  2. Citations:  10.8 points. [ EMJ 5y h-index is 36. Paper is 2 years old. 10/36 * 2/2 * 20 =5.6]
  3. Author: 4 points [ I’m not in the top 4 authors or senior author]
  4. TOTAL: 26.8 pts

This exercise for every academic could be done by one person with some coding skills.  I’m sure it could be calibrated to previous results and funding allocations by taking citations and papers for an earlier period. There may need to be tweaks to account for other kinds of academic outputs than just journal articles, but there are plenty of metrics available.

To summarise, I have just saved the country many millions of dollars and allowed academics to devote their time to what really matters.  All it needs now is for the decision makers to open their eyes and see the possibilities.

(ps. even easier would be to use the research component of the Times Higher Education World University Rankings and be done with it).

Advertisements

Cheesecake Files: The ICare-Acute Coronary Syndrome (heart attack) study

Hundreds of nurses, Emergency Department doctors, Cardiologists and other specialists, laboratory staff, administrators and managers from every hospital in New Zealand with an emergency department have come together to implement new, effective, and safe pathways for patients who think they may be having a heart attack.  Today, Dr Martin Than (CDHB, Emergency Department) presented to the American Heart Association results of our research into the national implementation of clinical pathways that incorporate an accelerated diagnostic protocol (ADP) for patients with possible heart attacks.  Simultaneously, a paper detailing that research is appearing in the academic journal Circulation.

The headlines, are that in the 7 hospitals we monitored (representing about 1/3rd of all ED admissions in NZ a year), there was a more than two fold increase in the numbers of patients who were safely discharged from the ED within 6 hours of arrival and told “It’s OK, you are not having a heart attack”.

Improving Care processes for patients with a possible heart attack.

Why is this important?

About 65,000 of the 1 million presentations to EDs each year in New Zealand are for patients whom the attending doctors think may be having a heart attack.  However, only 10-15% of those 65,000 are actually having a heart attack.  The traditional approach to assessment is long, drawn out, involves many resources, and means thousands of people are admitted into a hospital ward even thought it turns out they are not having a heart attack.  Of course, this means that they and their families have a very uncomfortable 24 hours or so wondering what is going on.  So, any method that safely helps to reassure and return home early some of those patients is a good thing.

What is a clinical pathway?

A clinical pathway is a written document based on best practice guidelines that is used by physicians to manage the course of care and treatment of patients with a particular condition or possible condition.  It is intended to standardise and set out the time frame for investigation and treatment within a particular health care setting – so it must take into account the resources available for a particular hospital.   For example, each hospital must document how a patient is assessed and if, for example, they are assessed within the ED as having a high-risk of a heart attack, where they must go.  In a large metropolitan hospital, this may mean simply passing them into the care of the cardiology department.  In a smaller setting like Taupo, where there is  no cardiology department, it may mean documenting when and how they are transported to Rotorua or Waikato hospital.

What is an accelerated diagnostic protocol?

An accelerated diagnostic protocol (ADP) is a component of the clinical pathway that enables the ED doctors to more rapidly and consistently make decisions about where to send the patient.  In all cases in New Zealand the ADPs for evaluating suspected heart attacks have 3 main components: (i) an immediate measurement of the electrical activity of the heart (an ECG), (ii) an immediate blood sample to look for the concentration of a marker of heart muscle damage called troponin, and a second sample 2 or 3 hours later, and (iii) a risk score based on demographics, prior history or heart conditions, smoking etc., and the nature of the pain (ie where it hurts and does it hurt when someone pushes on the chest, or when the patient takes deep breaths etc).   Importantly, these components enable a more rapid assessment of patients than traditionally and, in-particularly, enable patients to be rapidly risk stratified into low-risk, intermediate risk, and high-risk groups.  Usually the low-risk patients can be sent home.

What was done?

The Ministry of Health asked every ED to put in place a pathway.  Over an ~18 month period, a series of meetings were held at each hospital which were led by Dr Than, the clinical lead physician for the project.  Critically, at each meeting there were multiple members of the ED (doctors and nurses), cardiology, general wards, laboratory staff, and hospital administrators.  The evidence for different ADPs was presented.  Each hospital had to assess this evidence themselves and decide on the particularly ADP they would use.  Potential barriers to implementation and possible solutions were discussed.  Critically, champions for different aspects of the pathway implementation process were identified in each hospital.  These people led the process internally.

Oversight of the implementation was an adhoc advisory board put together by the Ministry of Health and with MoH officials, Dr Than, Cardiologists, and myself.

The Improving Care processes for patients with suspected Acute Coronary Syndrome (ICare-ACS) study was a Health Research Council sponsored study with co-sponsorship of staff time by participating hospitals.  Its goal was to measure any changes in each hospital to the proportions of patients who were being discharged home from ED early and to check whether they were being discharged safely (ie to check that there were not people with heart attacks were being sent home).  Dr Than and I co-led this project, but there were many involved who not only set up the pathways in each of the 7 participating study hospitals, but who also helped with attaining the data for me to crunch.

What were the study results?

In the pre-clinical pathway implementation phase (6 months for each hospital) there were 11,529 patients assessed for possible heart attack. Overall, 8.3% of them were sent home within 6 hours of arrival (we used 6 hours because this is a national target for having patients leave the ED).  The proportions of patients sent home varied considerably between hospitals – from 2.7% to 37.7%.  Of those sent home early, a very small proportion (0.52%) had what we call a major adverse event (eg a heart attack, a cardiac arrest, or death for any reason) within 30 days.  This is actually a very good number (it is practically impossible to be 0%).

We monitored each hospital for at least 5 months after pathway implementation and a median of 10.6 months.  Of the 19,803 patients, 18.4% were sent home within 6 hours of arrival.  ie the pathway more than doubled the number of patients who were sent home early.  Importantly, all 7 of the hospitals sent more patients home earlier.  The actual percentages sent home in each hospital still varied, showing there are more further improvements to be made in some hospital than others.  Very importantly, the rate of major adverse events in those sent home remained very low (0.44%).  Indeed, when we looked in detail at the few adverse events, in most cases there was a deviation from the local clinical pathway.  This suggests that some ongoing education and “embedding in” of the pathways may improve safety even more.

The study also showed that amongst all patients without a heart attack the implementation of the pathway reduced the median length of stay in hospital by nearly 3 hours.  Using crude numbers for the cost of an acute event in a hospital I estimate that this is a saving to the health system of $9.5Million per year.  These types are calculations are difficult and full of assumptions, nevertheless, I can be confident that the true savings are in the millions (pst… Government… I wouldn’t mind a fraction of this saving to carry on research please).

How did this come about?

This study and the pathway implementation is the result of a decade long series of studies in Christchurch hospital and some international studies, particularly with colleagues in Brisbane.  These studies have involved ED staff, cardiologists, research nurses, University of Otago academics (particularly those in the Christchurch Heart Institute) and many others.  They began with an international onbservational study which measured troponin concentrations at earlier than normal time points to see whether they gave information that would enable earlier discharge of some patients.  This was followed by the world’s first randomised trial of an ADP verse standard (then) practice.  That showed that the ADP resulted in more patients being safely sent home.  It was immediately adopted as standard practice in Christchurch.  The ADP was refined with a more “fit for purpose” risk assessment tool (called EDACS – developed locally and with collaboration of colleagues in Brisbane).  The EDACS protocol was then compared to the previous protocol (called ADAPT) in a second randomised trial.  It was at least as good with potential for discharging safely even more patients.  It is currently standard practice in Christchurch.

As a consequence of the Christchurch work, the Ministry of Health said, effectively,  ‘great, we want all of New Zealand to adopt a similar approach’, and the rest, as they say, is history.  Now, all EDs have a clinical pathway in place, all use an evidence based ADP – two use the ADAPT and all the rest use EDACS with one exception which uses a more ‘troponin centric’ approach (still evidence based) which I won’t go into here.  Meanwhile, all of Queensland has adopted the ADAPT approach and we know of many individual hospitals in Australia, Europe and Iran (yes) which have adopted EDACS.

Other help

As mentioned already, the Health Research Council and the Ministry of Health along with all those medical professionals were integral to getting to where we are today.  Also integral, were all those patients who in the randomised trials agreed to participate.  Medical research is build on the generosity of the patient volunteer.  Behind the scenes is our research manager, Alieke, who ensures doctors run on time.  Finally, I am very fortunate to be the recipient of a research fellowship that enables me to do what I do.  I thank my sponsors, the Emergency Care Foundation, Canterbury Medical Research Foundation, and Canterbury District Health Board.  Some of the earlier work has also been done in part with my University of Otago Christchurch hat on.  Thank you all.

A vision of kiwi kidneys

Sick of writing boring text reports.  Take a leaf out of Christchurch nephrologist Dr Suetonia Palmer’s (@SuetoniaPalmer) book and make a visual abstract report.  Here are two she has created recently based on data collected about organ donation and end stage renal failure by ANZDATA (@ANZDATARegistry). Enjoy.

Suetonia C-18RfJXUAApRcU

Suetonia C-16lBZXsAERoeM

ps. The featured image is of the Kidney Brothers.  Check out the great educational resources at The OrganWiseGuys.

How to improve your citation record

Peter Griffin over on Griffin’s Gadgets published a fun post on New Zealand’s seven most influential scientists based on data collected by Thomson Reuters and available at http://highlycited.com. Apparently they are all in the top 1% of cited scientists.  The ODT was obviously impressed by all this number waving and boasted of one of Dunedin’s own being part of the elite.  I was devestated not to be on that list, so I got thinking how I could move up the rankings.  Using Google scholar instead of Thomson Reuters is better for the ego of course because they allow a broader range of journals to be counted as citing or citable.  Unfortunately, if everyone did this I’d not be ranked any better.  Alternatively, I could send tweets out to everyone whom I cited hoping they’d be good enough to cite me back.  If I was really smart, I’d choose to cite most frequently those who publish most often.  Then I came across an easy answer in this graph – I must publish in Multidisciplinary journals!  I better get on with it, only 1650 potential citing days till PBRF 2018 …

Number of cites per document v H index for New Zealand documents published 2011-12. Source: SCImago. (2007). SJR — SCImago Journal & Country Rank. Retrieved June 25, 2014, from http://www.scimagojr.com

Number of cites per document v H index for New Zealand documents published 2011-12.
Source: SCImago. (2007). SJR — SCImago Journal & Country Rank.
Retrieved June 25, 2014, from http://www.scimagojr.com

 

Happy birthday “New Zealand Science Today”

It’s one year, 2184 articles, 460 subscribers, and 11,721 “flips” since the online Flipboard magazine which collates articles about what NZ scientists are doing and saying was first published.  Thanks to all the contributors, and especially to all those out in NZ Science-ville who are making a difference and letting the world know about it.  New Zealand Science Today can be found on Flipboard or on the internet here.NZ Science Today

 

 

100 days to do something about diabetes

So the NZ election is about 100 days away.  I want action from the political parties on an issue that in the next decade could affect a million of us, shortening lives, and cost us tens of billions of dollars.  The issue is simply diabetes.  Already 7% of adults have diabetes and another 18.6% is on the way to getting it (“pre-diabetic”). For our medical system  – and all tax payers – this means billions.  For individuals it means shortened lives, amputations, dialysis, blindness etc etc etc.  For employers it means workers taking sick days. For communities and families it means missing grandparents. Surely this is the biggest health issue and one of the biggest economic issues facing the country.  Where is the media about it?  Where are the questions to the politicians? I’ve blogged before about the lack of specific and evidence based policy amongst the political parties.  Where are their new policies?

Here’s a promise – I will publish on this blog any policy of any registered NZ political party specifically aimed to slow the diabetes epidemic. Along side that policy I’ll publish any evidence that is supplied as to why the party thinks that policy will work.

Free advertising – surely all the parties will take this up?

Science New Zealand

image

I’ve started a new Flipboard magazine called “Science New Zealand.”  Does anyone want to be a co-curator so we can collect news & commentary about NZ science & scientists in one place?

See

http://flip.it/6v48t