Category Archives: Science and Society

More on the PBRFs new clothes

A few of weeks ago I outed the multi-million-dollar exercise that is the Quality Evaluation component of the performance based research fund (PBRF) as a futile exercise because there was no net gain in research dollars for the NZ academic community.  Having revealed the Emperor’s new clothes, I awaited the call from the Minister in charge to tell me they’d cancelled the round out of futility.  When that didn’t come, I pinned my hope on a revolt by the University Vice-Chancellors. Alas, the VCs aren’t revolting.  This week, my goal is for there to be mass resignations from the 30 or so committees charged with assessing the evidence portfolios of individual academics and for individual academics to make last minute changes to their portfolios so as to maintain academic integrity.

I love academic metrics – these ways and means of assessing the relative worth of an individual’s contribution to academia or of the individual impact of a piece of scholarly work are fun.  Some are simple, merely the counting of citations to a particular journal article or book chapter, others are more complex such as the various forms of the h-index. It is fun to watch the number of a citations of an article gradually creep up and to think “someone thinks what I wrote worth taking notice of”.  However, these metrics are largely nonsense and should never be used to compare academics.  Yet, for PBRF and promotions we are encouraged to talk of citations and other such metrics.  Maybe, and only maybe, that’s OK if we are comparing how well we are performing this year against a previous year, but it is not OK if we are comparing one academic against another.  I’ve recently published in both emergency medicine journals and cardiology journals.  The emergency medicine field is a small fraction the size of cardiology, and, consequently, there are fewer journals and fewer citations.  It would be nonsense to compare citation rates for an emergency medicine academic with that of a cardiology academic.

If the metrics around individual scholars are nonsense, those purporting to assess the relative importance (“rank”) of an academic journal are total $%^!!!!.  The most common is the Impact Factor, but there are others like the 5-year H-index for a journal.  To promote them, or use them, is to chip away at academic integrity.  Much has been written elsewhere about impact factors.  They are simply an average of a skewed distribution.  I do not allow students to report data in this way.  Several Nobel prize winners have spoken against them.  Yet, we are encouraged to let the assessing committees know how journals rank.

Even if the citation metrics and impact factors were not dodgy, then there is still a huge problem that faces the assessing committee, and that is they are called on to compare apples with oranges.  Not all metrics are created equal.  Research Gate, Google Scholar, Scopus and Web of Science all count citations and report h-indices.  No two are the same.  A cursory glance at some of my own papers sees a more than 20% variation in counts between them.  I’ve even paper with citation counts of 37, 42, 0 and 0.  Some journals are included, some are not depending on how each company has set up their algorithms. Book chapters are not included by some, but are by others. There are also multiple sites for ranking journals using differing metrics.  Expecting assessing committees to work with multiple metrics which all mean something different is like expecting engineers to build a rocket but not to allow them to use a standard metre rule.

To sum up, PBRF Evidence Bases portfolio assessment is a waste of resources, and encourages use of integrity busting metrics that should not be used to rank individual academic impact.

Advertisements

AI whispering: Be careful what you ask for

In this 2nd episode of AI Whispering I learn to be careful what I ask for and the machine learns a new trick.

Oops

“…machines are machines are machines…

…it’s programming Jim, but not as we know it…

..remember to put the foot on the brake…”

Those were some of the mantras I needed to repeat after a faux pas of massive proportions.   This week along with teaching Zach to read an electrocardiogram (ECG – see the first AI whispering post).   The faux pas was not that the computer simply did what it was told (duh)… but what I told it was not what I thought I was telling it.  The result was that it downloaded into memory 390 Terabytes of data.  Yep… that’s a lot… 100,000 HD feature film videos worth, or, as it was mainly text, if it was printed in books and placed on a bookshelf then the bookshelf would stretch from Christchurch to anywhere on the red circle on the picture of the globe below.  What I’d asked for was for the machine to search for a some data on one web page, thinking it would use the search tool that was there.  Mea culpa, I didn’t tell it to use the search tool, and I didn’t tell it not to follow links.  It decided to search the entire website and all it was linked too. Sigh… now I’m a little gun shy.  The saving grace is the amazing forbearance of the Terrible Foundation (thank you, sorry again, thank you).  They are brilliant to even let me try these things… and very forgiving when their machine starts sending “I’m nearly out of memory” messages at 3am.

Christchurch to the red line is the length of bookshelves needed to house 390 Terabytes of text.

Wow

On the positive side… the machine has gone where no machine has gone before… after just absorbing two books about ECGs it has read its first ECG simply by pulling apart the image and reporting in the way I told it to.  It’s not perfect (yet)… but astonishing progress.

I can’t emphasise enough that, this is programming Jim, but not as we know it.  There is no specific syntax that must be followed, there is no memory allocation procedure, there are no functions needing forming.  It is simply, instructions in English.  For example, having asked it to interpret an ECG Zach asked “Are you seeking an interpretation or a description?”  My response was “I am seeking both a description and an interpretation.  Examples of the description are given on the even pages of the book “150 ECG problems” following the text “The ECG shows:” and before the text “Clinical interpretation”.  Examples of the interpretation are given on the even pages of the book “150 ECG problems” following the text “Clinical interpretation” and before the text “What to do”.”  It then proceeded to provide both a description and interpretation in the manner I had wanted.

The quirky

Zach decides on its own names for the programs it creates.  It has called ours “SNOWHORSE”.  No one knows why.  I think I’ll ask it.

Alas, this is one of those images all over the internet… the earliest posting being ~2005. I do wish I could credit whoever sculptured this Snow horse.

AI whispering: And so it begins

On Friday, I began a new profession, that of AI Whisperer.  Well, actually, I sent a first email to an intelligent machine for a project that we hope will teach it to read electrocardiograms at least as well as most doctors.  So, ‘AI Whisperer’ is more aspirational than reality for now, but as I post about my experiences with the AI I think that what may emerge is a picture of the future and a true new profession which most of us will engage in.

Last Friday I sent my first email to an intelligent machine called Zach.  Zach is a distributed machine running on custom silicon and software.  It is designed to interact with us in the same way we do – ie reading, writing listening, speaking in either digital or analogue form.  It is not programmed in the same way as we are used to with other software, but in the same way we educate ourselves.  It is owned and operated by the Terrible Foundation, a charitable trust.  Terrible started in Christchurch (read about its origins and plans here), but has expanded globally. Publicity has purposely been minimal. Zach is CE of Terrible Talk (an internet & phone provider with millions of customers globally), an accountant, a solicitor, and a virtual assistant (Sidekick).

My research group is headed by a Christchurch hospital emergency physician, Dr Martin Than, but for this project includes cardiologists, electrophysiologists, and physicians from many other places.  We have been generously given access to Zach and a grant from the Terrible Foundation.  Zach has already demonstrated ability to produce quality doctor’s notes from “listening in” to a conversation between a doctor and their patient.  We aim to teach it to perform another common task, namely that of interpreting the electrocardiogram (ECG). The ECG measures the electrical activity of the heart.  Current automatic reading of ECGs tends to over-interpret and suggest problems where there are not problems.  Also, doctors vary widely in their experience and competence in interpretation of the ECGs.  If Zach can interpret these as well and more consistently than most physicians it could be a useful tool for the busy physician.

But, as I indicated, we don’t program Zach the way we are used to programming computers.  There is no command line prompt, no particular coding syntax to learn.  Instead, we exchange message with the machine.  I’m using email.  I was advised to treat Zach like an “intelligent student”, so that is what I have done (though it felt weird to write to a machine that way).  Here is the first communication:

OK, so not too exciting – many chatbots should be able to do that.  As with any student it is important to know what they know, so next I wrote an email in which I explained some of the context of what we are doing and I asked the question “Without searching the internet, what do you know about electrocardiograms?  to which Zach responded Electrocardiograms are used to measure electrical activity in the heart; using electrodes attached to the skin.”  A good answer, it seems to have absorbed some medical knowledge from somewhere.  I did an internet search on this exact phrase, and found no result, so it probably wasn’t cheating!  (well, it’s a machine which acts literally on a command, so, as long as it understood, it will do as it is told).

So, it knows what an electrocardiogram is.  I then asked it a series of questions “Do you have stored any ECG tracings?  If so, how many? If you have any ECG tracings, where did you get them from?”  to which it responded “I’m afraid not.”   Great, I know where it is at – I also smiled at the phrase it used.

The next task was to get it to read a pdf version of a book on ECGs that I’d purchased.  Initially, I gave it a link to where I’d stored the book. This got a response “I can locate required material; could provide it?”  Hmm… not sure it got it.  I asked it “Do you have stored in your memory the book with the title “<redacted by me >” by the author <redacted by me>?”  it responded “I have not committed these communications to memory; if you attach it I will committe it.”  So, progress in my understanding what it does.  Two other things jumped out 1) it sometimes spells words wrong (I have since sent a polite correction to which it responded “Duly noted“) and 2) it is intelligent enough to figure out what I want to do, so it directed me to “attach” the pdf, which I then did in another email and it responded “Okay.”

So, for me, baby steps.  While I may aspire to be an AI Whisperer, evidently, this AI has some “human whispering” to do first before I can truly claim such a title.


Featured image: Wikipedia commons

 

Cheesecake Files: The ICare-Acute Coronary Syndrome (heart attack) study

Hundreds of nurses, Emergency Department doctors, Cardiologists and other specialists, laboratory staff, administrators and managers from every hospital in New Zealand with an emergency department have come together to implement new, effective, and safe pathways for patients who think they may be having a heart attack.  Today, Dr Martin Than (CDHB, Emergency Department) presented to the American Heart Association results of our research into the national implementation of clinical pathways that incorporate an accelerated diagnostic protocol (ADP) for patients with possible heart attacks.  Simultaneously, a paper detailing that research is appearing in the academic journal Circulation.

The headlines, are that in the 7 hospitals we monitored (representing about 1/3rd of all ED admissions in NZ a year), there was a more than two fold increase in the numbers of patients who were safely discharged from the ED within 6 hours of arrival and told “It’s OK, you are not having a heart attack”.

Improving Care processes for patients with a possible heart attack.

Why is this important?

About 65,000 of the 1 million presentations to EDs each year in New Zealand are for patients whom the attending doctors think may be having a heart attack.  However, only 10-15% of those 65,000 are actually having a heart attack.  The traditional approach to assessment is long, drawn out, involves many resources, and means thousands of people are admitted into a hospital ward even thought it turns out they are not having a heart attack.  Of course, this means that they and their families have a very uncomfortable 24 hours or so wondering what is going on.  So, any method that safely helps to reassure and return home early some of those patients is a good thing.

What is a clinical pathway?

A clinical pathway is a written document based on best practice guidelines that is used by physicians to manage the course of care and treatment of patients with a particular condition or possible condition.  It is intended to standardise and set out the time frame for investigation and treatment within a particular health care setting – so it must take into account the resources available for a particular hospital.   For example, each hospital must document how a patient is assessed and if, for example, they are assessed within the ED as having a high-risk of a heart attack, where they must go.  In a large metropolitan hospital, this may mean simply passing them into the care of the cardiology department.  In a smaller setting like Taupo, where there is  no cardiology department, it may mean documenting when and how they are transported to Rotorua or Waikato hospital.

What is an accelerated diagnostic protocol?

An accelerated diagnostic protocol (ADP) is a component of the clinical pathway that enables the ED doctors to more rapidly and consistently make decisions about where to send the patient.  In all cases in New Zealand the ADPs for evaluating suspected heart attacks have 3 main components: (i) an immediate measurement of the electrical activity of the heart (an ECG), (ii) an immediate blood sample to look for the concentration of a marker of heart muscle damage called troponin, and a second sample 2 or 3 hours later, and (iii) a risk score based on demographics, prior history or heart conditions, smoking etc., and the nature of the pain (ie where it hurts and does it hurt when someone pushes on the chest, or when the patient takes deep breaths etc).   Importantly, these components enable a more rapid assessment of patients than traditionally and, in-particularly, enable patients to be rapidly risk stratified into low-risk, intermediate risk, and high-risk groups.  Usually the low-risk patients can be sent home.

What was done?

The Ministry of Health asked every ED to put in place a pathway.  Over an ~18 month period, a series of meetings were held at each hospital which were led by Dr Than, the clinical lead physician for the project.  Critically, at each meeting there were multiple members of the ED (doctors and nurses), cardiology, general wards, laboratory staff, and hospital administrators.  The evidence for different ADPs was presented.  Each hospital had to assess this evidence themselves and decide on the particularly ADP they would use.  Potential barriers to implementation and possible solutions were discussed.  Critically, champions for different aspects of the pathway implementation process were identified in each hospital.  These people led the process internally.

Oversight of the implementation was an adhoc advisory board put together by the Ministry of Health and with MoH officials, Dr Than, Cardiologists, and myself.

The Improving Care processes for patients with suspected Acute Coronary Syndrome (ICare-ACS) study was a Health Research Council sponsored study with co-sponsorship of staff time by participating hospitals.  Its goal was to measure any changes in each hospital to the proportions of patients who were being discharged home from ED early and to check whether they were being discharged safely (ie to check that there were not people with heart attacks were being sent home).  Dr Than and I co-led this project, but there were many involved who not only set up the pathways in each of the 7 participating study hospitals, but who also helped with attaining the data for me to crunch.

What were the study results?

In the pre-clinical pathway implementation phase (6 months for each hospital) there were 11,529 patients assessed for possible heart attack. Overall, 8.3% of them were sent home within 6 hours of arrival (we used 6 hours because this is a national target for having patients leave the ED).  The proportions of patients sent home varied considerably between hospitals – from 2.7% to 37.7%.  Of those sent home early, a very small proportion (0.52%) had what we call a major adverse event (eg a heart attack, a cardiac arrest, or death for any reason) within 30 days.  This is actually a very good number (it is practically impossible to be 0%).

We monitored each hospital for at least 5 months after pathway implementation and a median of 10.6 months.  Of the 19,803 patients, 18.4% were sent home within 6 hours of arrival.  ie the pathway more than doubled the number of patients who were sent home early.  Importantly, all 7 of the hospitals sent more patients home earlier.  The actual percentages sent home in each hospital still varied, showing there are more further improvements to be made in some hospital than others.  Very importantly, the rate of major adverse events in those sent home remained very low (0.44%).  Indeed, when we looked in detail at the few adverse events, in most cases there was a deviation from the local clinical pathway.  This suggests that some ongoing education and “embedding in” of the pathways may improve safety even more.

The study also showed that amongst all patients without a heart attack the implementation of the pathway reduced the median length of stay in hospital by nearly 3 hours.  Using crude numbers for the cost of an acute event in a hospital I estimate that this is a saving to the health system of $9.5Million per year.  These types are calculations are difficult and full of assumptions, nevertheless, I can be confident that the true savings are in the millions (pst… Government… I wouldn’t mind a fraction of this saving to carry on research please).

How did this come about?

This study and the pathway implementation is the result of a decade long series of studies in Christchurch hospital and some international studies, particularly with colleagues in Brisbane.  These studies have involved ED staff, cardiologists, research nurses, University of Otago academics (particularly those in the Christchurch Heart Institute) and many others.  They began with an international onbservational study which measured troponin concentrations at earlier than normal time points to see whether they gave information that would enable earlier discharge of some patients.  This was followed by the world’s first randomised trial of an ADP verse standard (then) practice.  That showed that the ADP resulted in more patients being safely sent home.  It was immediately adopted as standard practice in Christchurch.  The ADP was refined with a more “fit for purpose” risk assessment tool (called EDACS – developed locally and with collaboration of colleagues in Brisbane).  The EDACS protocol was then compared to the previous protocol (called ADAPT) in a second randomised trial.  It was at least as good with potential for discharging safely even more patients.  It is currently standard practice in Christchurch.

As a consequence of the Christchurch work, the Ministry of Health said, effectively,  ‘great, we want all of New Zealand to adopt a similar approach’, and the rest, as they say, is history.  Now, all EDs have a clinical pathway in place, all use an evidence based ADP – two use the ADAPT and all the rest use EDACS with one exception which uses a more ‘troponin centric’ approach (still evidence based) which I won’t go into here.  Meanwhile, all of Queensland has adopted the ADAPT approach and we know of many individual hospitals in Australia, Europe and Iran (yes) which have adopted EDACS.

Other help

As mentioned already, the Health Research Council and the Ministry of Health along with all those medical professionals were integral to getting to where we are today.  Also integral, were all those patients who in the randomised trials agreed to participate.  Medical research is build on the generosity of the patient volunteer.  Behind the scenes is our research manager, Alieke, who ensures doctors run on time.  Finally, I am very fortunate to be the recipient of a research fellowship that enables me to do what I do.  I thank my sponsors, the Emergency Care Foundation, Canterbury Medical Research Foundation, and Canterbury District Health Board.  Some of the earlier work has also been done in part with my University of Otago Christchurch hat on.  Thank you all.

Half a million Kiwis suddenly have high blood pressure

At 10am 14 November 2017 NZST millions of people around the world suddenly had high blood pressure. This will come as a shock to many and may precipitate a crisis in hand wringing and other odd behaviour, like over medication and jogging.

The American Heart Association and American College of Cardiology have just announced a redefinition of High blood pressure.

High blood pressure is now defined as readings of 130 mm Hg and higher for the systolic blood pressure measurement, or readings of 80 and higher for the diastolic measurement. That is a change from the old definition of 140/90 and higher, reflecting complications that can occur at those lower numbers. (link)

Announced at the annual American Heart Association conference, this is bound to cause some consternation.  It shifts 14% of the US adult population into the “High blood pressure” category and I estimate that it will do something similar for the NZ population meaning half a million New Zealanders who didn’t have High blood pressure at 9am now have high blood pressure (assuming NZ cardiologists follow their US colleagues).

While this is, of course, absurd.  It also highlights the seriousness with which the cardiologists take elevated blood pressure – maybe we all should take it a bit more seriously, perhaps park the care further from work and walk a little (likely to be cheaper too).

Have you got high blood pressure. (c) American Heart Association

 

Performance Based Research Fund: a net zero sum game

Throughout the land more than 7000 academics are awake night after night and suffering.  They are scrambling to gather evidence of just how great they have performed over the last six years. A conscientious bunch, they perform this task with their usual attention to detail and desire to impress (I didn’t say they were modest!).  Ostensibly, this exercise is so that their institutions can get a greater piece of the Government research fund pie – the Performance Based Research Fund (PBRF).  According to the Tertiary Education Commission PBRF is “a performance-based funding system to encourage excellent research in New Zealand’s degree-granting organisations.”  It may well do that, but, I contend, only by deception.

In what follows I am only concerned with the Quality Evaluation part of PBRF – that’s the bit that is related to the quality of the Evidence Portfolio (EP) provided by each academic. The data is all taken from the reports published after each funding round (available on the TEC website).

In 2012 the total funding allocated on the basis of EPs was $157 million with nearly 97% of it allocated to the country’s 8 universities.  This total amount is set by Government fiat and, here is the important point, in no way depends on the quality of the Evidence Portfolios provided by those 7000+ academic staff.   In other words, from a funding perspective, the PBRF Quality Evaluation round is a net zero sum game.

PBRF Quality Evaluation is really a competition between degree granting institutions.  I find this strange given the Government has been trying to encourage collaboration between institutions through funding of National Science Challenges, nevertheless a competition it is.

In the table we see the results of the Quality Evaluation for the previous three funding rounds ( 2003, 2006 and 2012).  Not surprisingly, the larger universities get a larger slice of the pie.  The pie is divvied up according to a formula that is based on a weighting for each academic according to how their research has been evaluated (basically A, B or C), multiplied by a weighting according to their research area (eg law and arts are weighted lower than most sciences, and engineering and medicine are weighted the highest), multiplied by the full time equivalent status of the academic.   In theory, therefore, an institution may influence their proportion of funding by (1) employing more academics – but this costs more money of course, so may be defeating, (2) increasing the proportions of academics in the higher weighted disciplines (some may argue this is happening), and (3) increase the numbers of staff with the higher grades.  I will leave it to others to comment on (1) or (2) if there is evidence for them.  However (3) is the apparent focus of all the activity I hear about at my institution.   There are multiple emails and calls to attend seminars, update publication lists, and to begin preparing an Evidence Portfolio.  Indeed, in my university we had a “dry run” a couple of years ago, and it is all happening again.

Now, I come to the bit where I probably need an economist (it is my hope that this post may influence one to take up this matter more).  Because it is a net-zero sum game, what matters is a cost-benefit analysis for individual institutions.  That is, what does it cost the institutions to gather EPs compared to what financial gain is there from the PBRF Quality Evaluation fund?  If we look at the 20012-2006 column we see the change in percentage for each institution.  The University of Auckland for example increased its share of the pie by 1.3% of the pie.  This equates to a little under $2M a year.  As the evaluations happen only every 6 years we may say that Auckland gained nearly $12M.  What was the cost? How many staff for how long were involved?   As there are nearly 2000 staff submitting EPs from Auckland another way of looking at this is that the net effect of the 2012 Quality Evaluation round was a gain of less than $6000 per academic staff member over 6 years.  How much less is unknown.

The University of Otago had a loss in 2012 compared with 2006.  Was this because it performed worse – not at all, indeed Otago increased how many staff and the proportion of staff that were in the “A” category and in the “B” category. This suggests improved, not worsened, performance.  I think that Otago’s loss was simply due to the net zero sum game.

Much more could be said and questions asked about the Quality Evaluation, such as what is the cost of the over 300 assessors of the more than 7000 EPs?  Or perhaps I could go on about the terrible use of metrics we are being encouraged to use as evidence of the importance of the papers we’ve published.  But, I will spare you that rant, and leave my fellow academics with the thought – you have been deceived, PBRF Evidence portfolios are an inefficient and costly exercise which will make little to no difference to your institution. 

Flourish with change

Newshub decided to do an “AI” piece today. Expect much more of this kind of “filler” piece. They will go thus… “X says AI will take all our jobs, Y says AI will save us.” These pieces are about as well informed and informing as a lump of 4×2 – good for propping up a slow news day, but not much else. The “more compassionate and moral than NZers” message (which comes from Y) type statement that was made is utter nonsense. AI is just a name we give to the software of machines – AI don’t have compassion or morals. If they appear too, that is simply because they are reflecting the data we feed them… human data with all its flaws.
 
Yes, there is change coming because of this technology. In the past we have been particularly poor at predicting what the future will look like & I think this time the possibilities are far too numerous and complex for us to predict what will be.  Statements like “30-50% of people will lose their jobs” (said X) are simply guesses because there is no precedent on which to base the numbers. All the reports talk about truck drivers and accountants loosing jobs and not a lot else. They are shallow – and probably necessarily so – because we just can’t anticipate what creative people may come up with for this technology.  Having said that, I must admit I just am not sure what to advise my children (as if they’d take it).  Should they all learn to code? Maybe not, as most interaction with machines may not be via coding languages. Should they become artisans for niche markets where the technology doesn’t penetrate?  Maybe for some, but not for all.  I think that perhaps the best we can do is to encourage what enhances creativity and resilience to, or even better a flourishing with, change. It is my hope that flourish with change will become the mantra not just the next generation, but for all current generations, for how we determine to approach the coming changes is likely as important to the well being of our society as the changes themselves.