Category Archives: $100Dialysis

More on the PBRFs new clothes

A few of weeks ago I outed the multi-million-dollar exercise that is the Quality Evaluation component of the performance based research fund (PBRF) as a futile exercise because there was no net gain in research dollars for the NZ academic community.  Having revealed the Emperor’s new clothes, I awaited the call from the Minister in charge to tell me they’d cancelled the round out of futility.  When that didn’t come, I pinned my hope on a revolt by the University Vice-Chancellors. Alas, the VCs aren’t revolting.  This week, my goal is for there to be mass resignations from the 30 or so committees charged with assessing the evidence portfolios of individual academics and for individual academics to make last minute changes to their portfolios so as to maintain academic integrity.

I love academic metrics – these ways and means of assessing the relative worth of an individual’s contribution to academia or of the individual impact of a piece of scholarly work are fun.  Some are simple, merely the counting of citations to a particular journal article or book chapter, others are more complex such as the various forms of the h-index. It is fun to watch the number of a citations of an article gradually creep up and to think “someone thinks what I wrote worth taking notice of”.  However, these metrics are largely nonsense and should never be used to compare academics.  Yet, for PBRF and promotions we are encouraged to talk of citations and other such metrics.  Maybe, and only maybe, that’s OK if we are comparing how well we are performing this year against a previous year, but it is not OK if we are comparing one academic against another.  I’ve recently published in both emergency medicine journals and cardiology journals.  The emergency medicine field is a small fraction the size of cardiology, and, consequently, there are fewer journals and fewer citations.  It would be nonsense to compare citation rates for an emergency medicine academic with that of a cardiology academic.

If the metrics around individual scholars are nonsense, those purporting to assess the relative importance (“rank”) of an academic journal are total $%^!!!!.  The most common is the Impact Factor, but there are others like the 5-year H-index for a journal.  To promote them, or use them, is to chip away at academic integrity.  Much has been written elsewhere about impact factors.  They are simply an average of a skewed distribution.  I do not allow students to report data in this way.  Several Nobel prize winners have spoken against them.  Yet, we are encouraged to let the assessing committees know how journals rank.

Even if the citation metrics and impact factors were not dodgy, then there is still a huge problem that faces the assessing committee, and that is they are called on to compare apples with oranges.  Not all metrics are created equal.  Research Gate, Google Scholar, Scopus and Web of Science all count citations and report h-indices.  No two are the same.  A cursory glance at some of my own papers sees a more than 20% variation in counts between them.  I’ve even paper with citation counts of 37, 42, 0 and 0.  Some journals are included, some are not depending on how each company has set up their algorithms. Book chapters are not included by some, but are by others. There are also multiple sites for ranking journals using differing metrics.  Expecting assessing committees to work with multiple metrics which all mean something different is like expecting engineers to build a rocket but not to allow them to use a standard metre rule.

To sum up, PBRF Evidence Bases portfolio assessment is a waste of resources, and encourages use of integrity busting metrics that should not be used to rank individual academic impact.

Advertisements

AI whispering: Be careful what you ask for

In this 2nd episode of AI Whispering I learn to be careful what I ask for and the machine learns a new trick.

Oops

“…machines are machines are machines…

…it’s programming Jim, but not as we know it…

..remember to put the foot on the brake…”

Those were some of the mantras I needed to repeat after a faux pas of massive proportions.   This week along with teaching Zach to read an electrocardiogram (ECG – see the first AI whispering post).   The faux pas was not that the computer simply did what it was told (duh)… but what I told it was not what I thought I was telling it.  The result was that it downloaded into memory 390 Terabytes of data.  Yep… that’s a lot… 100,000 HD feature film videos worth, or, as it was mainly text, if it was printed in books and placed on a bookshelf then the bookshelf would stretch from Christchurch to anywhere on the red circle on the picture of the globe below.  What I’d asked for was for the machine to search for a some data on one web page, thinking it would use the search tool that was there.  Mea culpa, I didn’t tell it to use the search tool, and I didn’t tell it not to follow links.  It decided to search the entire website and all it was linked too. Sigh… now I’m a little gun shy.  The saving grace is the amazing forbearance of the Terrible Foundation (thank you, sorry again, thank you).  They are brilliant to even let me try these things… and very forgiving when their machine starts sending “I’m nearly out of memory” messages at 3am.

Christchurch to the red line is the length of bookshelves needed to house 390 Terabytes of text.

Wow

On the positive side… the machine has gone where no machine has gone before… after just absorbing two books about ECGs it has read its first ECG simply by pulling apart the image and reporting in the way I told it to.  It’s not perfect (yet)… but astonishing progress.

I can’t emphasise enough that, this is programming Jim, but not as we know it.  There is no specific syntax that must be followed, there is no memory allocation procedure, there are no functions needing forming.  It is simply, instructions in English.  For example, having asked it to interpret an ECG Zach asked “Are you seeking an interpretation or a description?”  My response was “I am seeking both a description and an interpretation.  Examples of the description are given on the even pages of the book “150 ECG problems” following the text “The ECG shows:” and before the text “Clinical interpretation”.  Examples of the interpretation are given on the even pages of the book “150 ECG problems” following the text “Clinical interpretation” and before the text “What to do”.”  It then proceeded to provide both a description and interpretation in the manner I had wanted.

The quirky

Zach decides on its own names for the programs it creates.  It has called ours “SNOWHORSE”.  No one knows why.  I think I’ll ask it.

Alas, this is one of those images all over the internet… the earliest posting being ~2005. I do wish I could credit whoever sculptured this Snow horse.

Flourish with change

Newshub decided to do an “AI” piece today. Expect much more of this kind of “filler” piece. They will go thus… “X says AI will take all our jobs, Y says AI will save us.” These pieces are about as well informed and informing as a lump of 4×2 – good for propping up a slow news day, but not much else. The “more compassionate and moral than NZers” message (which comes from Y) type statement that was made is utter nonsense. AI is just a name we give to the software of machines – AI don’t have compassion or morals. If they appear too, that is simply because they are reflecting the data we feed them… human data with all its flaws.
 
Yes, there is change coming because of this technology. In the past we have been particularly poor at predicting what the future will look like & I think this time the possibilities are far too numerous and complex for us to predict what will be.  Statements like “30-50% of people will lose their jobs” (said X) are simply guesses because there is no precedent on which to base the numbers. All the reports talk about truck drivers and accountants loosing jobs and not a lot else. They are shallow – and probably necessarily so – because we just can’t anticipate what creative people may come up with for this technology.  Having said that, I must admit I just am not sure what to advise my children (as if they’d take it).  Should they all learn to code? Maybe not, as most interaction with machines may not be via coding languages. Should they become artisans for niche markets where the technology doesn’t penetrate?  Maybe for some, but not for all.  I think that perhaps the best we can do is to encourage what enhances creativity and resilience to, or even better a flourishing with, change. It is my hope that flourish with change will become the mantra not just the next generation, but for all current generations, for how we determine to approach the coming changes is likely as important to the well being of our society as the changes themselves.

This is what happens when you talk to your mother about artificial intelligence

Artificial Intelligence 

Artificial Intelligence

So we don’t need to think.

Everything is done for us

In just an eyelid blink.

 

Artificial Intelligence

So we don’t need to think.

Just take the Robot, plug it in

And go and have a drink.

 

When you come back your work is done;

You haven’t even thunk.

The Robot’s done the washing too;

Oh dear, I think it’s shrunk.

 

Perhaps I shouldn’t have bought this one,

I didn’t even think,

I got it second hand you see

From prisoners in the clink.

 

And when they programmed it you see

I think that they were drunk

‘Cos now it’s full of nasty words;

I really should have thunk.

 

So artificial Intelligence

Depends upon the thought

That someone programmes into it,

And that may come to nought.

 

And so beware when buying one,

You may be feeling sunk,

It may be right for it to think,

But you also should have thunk!

(c) K.A. Pickering, October 2017

AI Robot copy (1)

Artificial Intelligence (c) K.A. Pickering, October 2017

The wrong impact

“We just got a paper in an Impact Factor 10 journal … and hope to go higher soon.”  That’s a statement made to me last week.  It is wrong on so many levels, but does it matter?   Nobel Prize winners think so. This video from nobelprize.org appeared in my twitter feed on Friday.  Before you watch it, consider this, academics in NZ are being encouraged in promotion applications and in preparing for the next round of NZ Performance Based Research Fund (PBRF), which will allocate millions of dollars to academic institutions, to include a metric of the ranking of the journal.  The Impact Factor is the most common metric available.

 

ps. I would not allow a student working with me to present a raw mean of a highly skewed distribution because it so very poorly represents the distribution.  However, this is exactly what the Impact Factor does (for those who don’t know the most common impact factor for a journal in any given year is simply the sum of citations of articles from the preceding two years divided by the total number of articles published.  The citation distribution is usually skewed because the vast majority of articles receive very few citations in such a short time, but a few receive a lot).  There are numerous other problems with it, not the least that it can’t be used to compare “impact” between different disciplines.

A vision of kiwi kidneys

Sick of writing boring text reports.  Take a leaf out of Christchurch nephrologist Dr Suetonia Palmer’s (@SuetoniaPalmer) book and make a visual abstract report.  Here are two she has created recently based on data collected about organ donation and end stage renal failure by ANZDATA (@ANZDATARegistry). Enjoy.

Suetonia C-18RfJXUAApRcU

Suetonia C-16lBZXsAERoeM

ps. The featured image is of the Kidney Brothers.  Check out the great educational resources at The OrganWiseGuys.

An even quicker way to rule out heart attacks

The majority of New Zealand emergency departments look for heart muscle damage by taking a sample of blood and looking for a particular molecule called a high-sensitivity troponin T (hsTnT).  We have now confirmed that rather than two measurements over several hours just one measurement on arrival in the ED could be used to rule out heart attacks in about 30% of patients.

What did we do?

We think this is a big deal. We’ve timed this post to meet the Annas of Internal Medicine timing for when our work appears on their website – here.  What we did was to search the literature to find where research groups may have measured hsTnT in the right group of people – namely people appearing in an emergency room whom the attending physician thinks they may be having a heart attack. We also required that the diagnosis of a heart attack, or not, was made not by just one physician, but by at least two independently.  In this way we made sure we were accessing the best quality data.

Next I approached the authors of the studies as asked them to share some data with us – namely the number of people who had detectable and undetectable hsTnT (every blood test has a minimum level below which it is said to be “undetectable” in hsTnT’s case that is just 5 billionths of a gram per litre, or 5ng/L).  We also asked them to check in these patients if the electrical activity of the heart (measured by an electrocardiogram or “ECG”) looked like there may or may not be damage to the heart (a helpful test, but not used on its own to diagnose this kind of heart attack).  Finally, we asked the authors to identify which patients truly did and did not have a heart attack.

What did we find?

In the end research groups in Europe, UK, Australia, NZ, and the US participated with a total of 11 studies and more than 9000 patients.  I did some fancy statistics to show that overall about 30% of patients had undetectable hsTnT with the first blood test and negative ECGs.  Of all those who were identifiable as potentially “excludable” or “low-risk” only about 1 in 200 had a heart attack diagnosed (we’d like it to be zero, but this just isn’t possible, especially given the diagnosis is not exact).

VisualAbstract AnnalsIM 170411

Pickering, J. W.*, Than, M. P.*, Cullen, L. A., Aldous, S., Avest, ter, E., Body, R., et al. (2017). Rapid Rule-out of Myocardial Infarction With a High-Sensitivity CardiacTroponin T Measurement Below the Limit of Detection: A Collaborative Meta-analysis. Annals of Internal Medicine, 166(10). http://doi.org/10.7326/M16-2562 *joint first authors.

What did we conclude?

There is huge potential for ruling out a heart attack with just one blood test.  In New Zealand this could mean many thousands of people a year can be reassured even more swiftly that they are not having a heart attack. By excluding the possibility of a heart attack early, physicians can put more effort into looking for other causes of chest-pain or simply send the patient happily home.   While not every hospital performed had the same great performance, overall the results were good.  By the commonly accepted standards, it is safe.  However, we caution that local audits at each hospital that decides to implement this “single blood measurement” strategy are made to double check its safety and efficacy.


Acknowledgment: This was a massive undertaking that required the collaboration of dozens of people from all around the world – their patience and willingness to participate is much appreciated. My clinical colleague and co-first author, Dr Martin Than provided a lot of the energy as well as intelligence for this project. As always, I am deeply appreciative of my sponsors: the Emergency Care Foundation, Canterbury Medical Research Foundation, Canterbury District Health Board, and University of Otago Christchurch. There will be readers who have contributed financially to the first two (charities) – I thank you – your generosity made this possible, and there will be readers who have volunteered for clinical studies – you are my heroes.

Sponsors