Cheesecake Files: Machine learning heart attacks

“Machine learning” rates very high on the buzz-word scale, right up there with “nano-technology” and “blockchain”. Like most buzz it is more noise than substance. However, every now and again it looks like there might be something in the noise that bites. This episode of the Cheesecake Files1 is about testing an algorithm (another buzz word) developed through a machine learning technique for the early detection of heart attacks (strictly – myocardial infarction).

The buzz

Before I begin my story in earnest a couple of words about the buzz words. When I say “algorithm” think “recipe”. In the context of emergency medicine this is simply a series of steps which assist the medical team in their decision making. For example – if the presenting complaint is “chest pain” the triage nurse will connect up a device (ECG) to measure the electrical activity of the heart and will draw some blood and send it off the the lab with specific instructions to measure the concentration of a molecule called troponin. Several years ago we introduced in all New Zealand emergency departments more detailed pathways (ie an algorithm) which included guidance on which other data to obtain from the patient, when to repeat blood measurements and how all the data goes together to risk stratify the patient. The principal aim being to ensure that as quickly as possible, and as safely as possible, physicians could rule out the presence of a heart attack. This is important because patients presenting with possible heart attacks are one of the most common presentations to the ED and so if they remain in the ED a long time this can affect the whole service. However, only about 10-15% (in NZ) are actually having a heart attack. Many of those who aren’t can now be reassured early that they are not. Please note – if you’ve sudden onset chest pain then the ED is the right place for you. Just because most who attend are not having a heart attack doesn’t mean that you might not be.

The other buzz word is “machine learning.” This term is usually used to mean a computational technique which involves giving a computer some data and some basic instructions how to look at it. Then asking the computer to make a prediction of an outcome (in our case, whether a patient is having a heart attack or not). The prediction is compared to the actual outcomes and information on how well the computer performs is feedback into the machine to tweak some of the algorithm. Think of this as tasting the soup and then adding a few more spices. The process is repeated many times until the soup is as good as it can possibly be. Some recipes we know and can follow ourselves. Some happen behind closed doors as a team of chefs puts together a meal. A characteristic of machine learning algorithms is that they are often not easily understood (a “black-box”), but the proof of the pudding is in the eating. This leads me to the story that is the current cheesecake.

The story

Nearly three years ago we were asked to test if an algorithm called MI3 works to risk stratify people who appear in the emergency department with symptoms suggestive of a heart attack. The algorithm had been developed by a US based diagnostic company called Abbott Diagnostics. We were given access to the black box and could input variables from real patients and observe the predicted outcome. In this case the algorithm was producing a number that very closely corresponded to the probability of a patient having a heart attack. There were very few variables required to make this prediction – sex, age, two measures of troponin and the time between the two measures. The latter is important because how troponin concentrations change over time informs us about the possible heart attack.

A collaboration of research groups from Scotland, Switzerland, Germany, United States of America, Australia and New Zealand came together to provide sufficient data to test MI3. This group was lead by Christchurch ED physician Dr Martin Than, and Scottish cardiologist, Prof Nicholas Mills. I was charged with pulling together all the data and conducting the statistical analysis of the performance of MI3.

There were about 8000 patients in our testing data set with 10.6% of them having a heart attack. Importantly, the first thing I noted is that the values output by the algorithm corresponded to the true rate of heart attacks. ie when the MI3 value was 5 about 5% of those with this value were having a heart attack, when it was 90 about 90% of people were having a heart attack. In other words, the algorithm was well calibrated – this can give physicians confidence. The second thing was to see if we could find MI3 values below which we could say that almost everyone is not having a heart attack (it’s impossible to be 100% certain – we aim for about 99% or better). We were able to find such a value and show that it identified an impressive 69% of people as low-risk. The full results are available in the cardiology journal Circulation – here.

The application

So, how may this be used? The difference with this algorithm compared with others is three-fold (i) it does not require blood samples to be taken an specific set intervals, (ii) it does not require information about patient history or detailed signs and symptoms to be gathered and incorporated, (iii) and the output is a probability rather than simply stratifying patients to a low, intermediate or high risk category. In other words, the inputs are simple and objective, and the output is easily interpretable. In practice, the physician may receive the MI3 value from the labs along with the troponin results. This may aid discussions with the patient through the use of icon arrays or similar (see the figure).

A concept of how a tool displaying the result of the algorithm may be used to display risk to physician and patient.

1 Once upon a time, a long long time ago, I received a cheesecake for every publication. Sadly, those days are gone now. But I live in hope.

Disclaimer: I have acted as a consultant statistician for Abbott Diagnostics. I have no shares or intellectual property associated with MI3. Abbott was not involved in the testing of the algorithm.

Dark space

Dark space, like green space, is essential for our well being. Dark space, like green space, is our past, our taonga, and our right. Dark space, unlike green space, is not prioritised in our city plans, is not part of our conversation about Te Tiriti, nor is it where we go for relaxation and inspiration. But once it was. Dark space is accessible to us all, if only. If only we turned off the lights and looked up. Dark space is the night sky, the moon, the planets, the stars and galaxies. Once visible to every child, now lost in the haze of light pollution; once the source of wonder and joy; once the inspiration and the starting point for the personal journeys of countless scientists and philosophers, religious leaders and poets.

The week is Matariki. The new year celebrated at the first appearance of the stars known as Matariki – the pleiades. A time for friends and family and celebration. Sadly, even tragically and certainly scandalously most tamariki in Aotearoa will struggle to see any of the stars of Matariki, let alone the nine that are supposedly naked eye objects. This is because we have not acted as kaitiaki of our night sky. We have not guarded it from our modern obsession of pretending night is just an extension of day rather than a time to rest, recuperate, and, yes, gaze on the heavens in awe-filled wonder.

Matariki (Source: http://deography.com/m45-the-pleiades-seven-sisters/)

All is not lost, though. The solutions are in our own hands. As we have preciously defended the green spaces in our city, now we must do the same for dark space. We must insist that our city ordinances are such that lighting that spills light into the dark space is not permitted and that the kinds of lights we employ are not of the most polluting and disrupting kinds. As individuals we can turn down or off our outside lights that blaze away even when we are not there. We must be guardians of the night – protecting not just the taonga that is the view, but the darkness that enables our night loving insects and birds to survive. Matariki is also a time to reflect on our health and wellbeing. Perhaps this year we may reflect on the fact that the blue light emitted by our phones and tablets which we have been warned is detrimental to our health, is the same kind of light emitted by the LED street lights, which cities are rushing to pollute dark space with.

Let us together, all who call Aotearoa New Zealand home, reclaim our kaitiakitanga of Matariki.

(Featured Image by cafuego https://www.flickr.com/photos/cafuego/32719827268)

Performance Based Research Fund: The numbers are up

8269 academics and their bosses have been alerted – the data is in, the numbers are crunched, PBRF scores are out. Who are the winners, who are the losers? Find out more with tec publication. But before you go there…

I predict that any minute now tertiary institutions throughout the land will be posting press releases detailing their successes, each one trying to say “we’re the best” – at least in some category, somehow, if you squeeze the numbers and look at them sideways… well, you get the picture. Speaking of which – here’s one I posted after the last lot of PBRF results were released in 2013.

http://www.scoop.co.nz/education 12 April 2013

Last year I stated that the PBRF quality evaluation was a net zero sum game, because irrespective of the outcomes of the quality evaluations of the 8269 staff there is no net increase in the overall funding (that being set by the Government of the day in the Budget). The total quality evaluation dollar amounts are $173,250,000 p.a. The 2018 PBRF process has shifted where this pie is divided up marginally.

The percentage changes are marginal, but they do translate to a loss or gain to individual institutions. In this round, based on these preliminary results, the “big winners” appear to be AUT, VUW and the non-university sector. The biggest loser appears to be UC, but the others seem to have lost a substantial amount. Of course, as I’ve argued before, these $ amounts must be considered in terms of the total $ costs to the institutions and the government to administer the PBRF, likely millions for each institution. The gains of VUW and AUT aren’t really as much as they look and the losses to the other institutions more.

Was this Quality Evaluation PBRF process worthwhile? – I think not. Will the results be celebrated? – I expect so. Will this post do more to expose the emperor’s new clothes and change the system? – one lives in hope.

Falling into a black hole

To honour Roy Kerr and the first picture of a Black Hole, I have used the wonderful “rayshader” package in R to render the photo into 3D. It’s a bit like an extraordinarily hot volcano. Don’t fall in.

Thanks NASA for the photo (https://www.nasa.gov/mission_pages/chandra/news/black-hole-image-makes-history).

Is my science ethical?

Am I doing what the public thinks is OK to do?  That was the question that came to mind as I heard Bioethics and Health Law expert Rochelle Style speak at a Health data workshop held at the University of Otago Christchurch this week.  Rochelle spoke with clarity and demonstrated a great deal of expertise.  I like the simplicity of Law – tells us what we can do, Ethics – tells us what we should do, and Social license – tells us what the publish think is OK to do.  In this post I want to focus on Social license because when I asked a question about it, it became evident that this was the least understood or investigated of the three legs to ethical decision.

First, an example.  It is legal for me to search twitter for posts about prostate cancer.  If I was to approach the NZ Health and Disabilities Ethics Communities saying that I was doing an observational study of publicly available social media comments on prostate cancer I may not even need to go for ethical review.  However, would it be OK – according to the NZ public – for me to do this?  What if in my report I anonymously quoted some of those twitter posts?  What if I quoted some of those twitter posts and identified the person who made the post?

What is your reaction?  Which is OK?

Now consider this, the print and online media use identified twitter quotes all the time.  They effectively take what is public in one domain to an audience that chooses to follow someone and make it public to another audience who has not made that choice. In addition, consider that twitter uses can block certain people from seeing their tweets.  This suggests to me that the “social license” employed by the media is quite broad. Is it OK?

What if when you visited your doctor you were told by the receptionist that the consultation may be recorded for training purposes.  There was no option to “opt out”.  Would this be OK?  Now, phone your bank or broadband provider.  Did you get an option to “opt out”?  They now have your voice recorded – a biometric that can be used to identify you – as well as details of the conversation.  How long will they keep it for?  Who can listen to it? How is the data stored? How can you ask for it to be deleted?

Hopefully, by now, you see that the concept of social license – what is OK and what isn’t – is not easy to tie down.  For health science, trying to understand what the public think is OK and what they think is not OK is very important, though not well articulated.  

Two organisations that are actively engaged in the discussion about the use of social license for health (and other) data are Te Mana Raraunga , the Māori Data Sovereignty Network and Data Futures Partnership.  Their websites are worth visiting.  Te Mana Raraunga talk of data as “taonga” (a “treasure” for a non-New Zealander reader). This really appeals to me and I think is a good place to start, whether as individuals or communities.  

If our personal data is taonga, what are our rights and responsibilities over this taonga?  How we answer this will depend on our own cultural, social, religious, and other backgrounds.  For me, my data is taonga and I have a responsibility to use this treasure for the greater good – the greater good of those around me, and for generations to come.  This is one reason I volunteer for health studies from time to time.

Regarding data as taonga belonging to individual patients helps me in my role as a scientist. It is not just about making sure I don’t inadvertently reveal something publicly about an individual, but that I utilise ethically and well the data that I have been given temporary guardianship over. For me, this means that I must be vigilant over the statistical techniques I use and cognisant of the most appropriate ways to interpret that data.  Unfortunately, this is a much-neglected area of scientific ethics.  The scientific literature, even in the most prestigious of journals, is full of error of interpretation and of statistical methodology. I regularly see even simple errors in journal articles I referee.  I know I have committed some errors of these myself.  What I believe is most important is that I take responsibility for improving the techniques I use and I improve how I interpret and communicate the results. This is not just about “better science” but about treating the data as taonga, about regarding myself as guardian, not owner, of the data, and recognising that I operate not merely under law and ethical guidelines, but under a social license.  That license may be fluid, ill-defined, and somehow based on diverse views, but nevertheless it exists and needs to be understood and honoured.

The physics of maiming a child (repost because of “those” scooters)

Dear Driver,

When you backed out of a driveway and did not even see how I swerved around behind your car to avoid T-boning you, how dare you have the temerity to tell me you were careful!  I was 7 feet tall, dressed in bright yellow and traveling at no more than 10 km/h.  Perhaps a simple lesson in physics will help you and your fellow “driveway backers” to realise how dangerous you are and to adopt safer driving practices.

In the diagram you can see a car backing out of a driveway.  Typically when you are at the edge of your property and have a fence (see photo below) blocking your view of the footpath you are able to see about 1.7 metres along the footpath.  Let us imagine that there is a child on a trike riding at 5 km/h just out of your line of sight.  How long  does it take them to travel that 1.67 metres?  The physics is quite easy.

5 km/h is 5000 metres in 60 x 60 seconds, ie about 1.4 m/s.  Putting this in the formula above means that it takes about 1.2 seconds for the child to travel that 1.67 metres. 

Now consider this. According to design guidelines for safe bicycle use 2.5 seconds must be allowed for someone to observe the danger, react, apply brakes and stop.  In other words, if you covered the distance from your driveway to the middle of the footpath, about 1 metre, in under 1.2 seconds you will almost certainly hit the child.  That is a speed of just 3 km/h!!!!!

Now consider who else is on the footpath, all legally:

  • Pedestrians 5 km/h
  • Joggers 5- 15 km/h
  • Kids on skateboards or scooters 10 km/h
  • Child on bicycle with small wheels, 10 km/h
  • Mobility scooter, 5-10 km/h
  • Me on my Trikke, 10 km/h
  • Postie on a bike 5-10 km/h.

For those going 10 km/h your speed needs to be just over 1.5 km/h to hit someone! That’s the legal people … but the Lime scooters at 20 km/h mean it all the more necessary to slow down.

So, before you do some damage here is what you can do:

  • Never back out of a driveway unless you really really must.  If you think you must because of the design of your driveway, change the design!
  • Cut back those hedges, remove some of that fence so that you can see further. [ City councils… please make a by-law to make this happen].
  • Always always always stop at the end of your driveway (BEFORE THE FOOTPATH) and toot a horn.  Then proceed very very slowly.

By the way, you are legally obliged to give way:

(1)
A driver entering or exiting a driveway must give way to a road user on a footpath, cycle path, or shared path (as described by clause 11.1A(1)).

Thank you for considering the physics of maiming a child, may you never find your self in such a terrible situation.

Regards,

Dr John Pickering

A typical driveway with almost non-existant visibility
 A typical driveway with almost non-existent visibility

———

Feature Image: Intangible Arts https://www.flickr.com/photos/intangible/ under Creative Commons Attribution 2.0 licence.

Cheesecake files: A new test to rule out heart attacks in just a few minutes.

Your chest hurts, you go to the hospital (good move), you get rushed through and a nurse takes some blood and measures the electrical activity of your heart.  A doctor asks you some questions.  While she does so, the blood is being tested – the results are back already! Yeah, they are negative and everything else is OK, it’s not a heart attack – you can go home.  This is the likely scenario in the near future thanks to new blood test technology which we, in Christchurch hospital’s Emergency Department, have been fortunate to be the first in the world to trial in patients. The results of our pilot study have now been published ( in a Journal of the American Medical Association (JAMA Cardiology).

About 65,000 patients a year are investigated for heart attacks in New Zealand emergency departments, yet only about 15% of them are actually having a heart attack.  New Zealand leads the world in having become the first country in the world in which all patients are assessed by an accelerated diagnostic pathway that enables rapid evaluation of the patients and can send people home after two blood tests taken two to three hours apart (see here for more).  This means many patients who once-upon-a-time would have been admitted to hospital overnight, are now able to be reassured after 4-6 hours that they are not having a heart attack and can go home.  Nevertheless, there are enormous advantages for both patient and health system to being able to come to the conclusion that the pain isn’t life threatening earlier. The cork in the bottle preventing this happening is the time it takes for a blood sample to be analysed for signs of damage to the heart. These blood tests typically take 1 to 2 hours from the time of sampling (within ~15 minutes of arrival in the ED) until the results are available for the doctor to review.  Because doctors are dealing with multiple patients at a time, their review and decisions around whether to allow the patient to go home, or to be admitted for more investigation, are further delayed.  A point-of-care test is one that happens with a small machine near the bedside and can produce results available to the doctor even while they are still examining the patient.  Until now, though, the precision of these machines has not been good enough to be used in emergency departments.  When one manufacturer told us that their new technology may now have sufficient precision we were keen to test it,  so we, in a first-in-the-world study, undertook a study in patients entering the emergency department of Christchurch hospital whom the attending doctor was investigating for a possible heart attack.

Thanks to the volunteer patients (I love volunteers) who gave some extra blood we measured the troponin concentration by this new point-of-care test (called the next generation point of care troponin I: TnI-Nx). Troponin comes from the heart muscle and is released into the blood during a heart attack. When the troponin concentrations in the blood are very very low we can be confident that the source of the patient’s discomfort is not a heart attack.  Low concentrations require a very precise measurement test. Often, a very low concentration means the patient can safely go home. In 354 volunteers we measured troponin with the TnI-Nx assay when they first came to the emergency department.  Their treatment didn’t change, and all clinical decisions were based on the normal laboratory based troponin (measured on entry to the emergency department and again 2 hours later). From the blood samples we collected and measurements we made, we could work out what could have happened if we had used the TnI-Nx results instead.

In our study the TnI-Nx troponin measurement was as good as, and possibly slightly better, than the laboratory based troponin measurement at ruling-out a heart attack. We found 57% of the patients being investigated had troponin concentrations measured with TnI-Nx below a threshold at which we could be confident that they were not having a heart-attack.  All 57 patients who were actually having a heart attack had higher concentrations.

When implemented our results may mean that instead of waiting 3-6 hours for a results, half of patients being investigated could know within about 30 minutes of arriving at the ED whether they are having a heart attack or not.  This early reassurance would be a relief to many, as well as reducing over-crowding in the emergency department and freeing up staff for other tasks.  But before we implement the new test, we must validate it in more patients – this is a study we are carrying out now.  Validation will enable us to more precisely determine a threshold concentration for TnI-Nx for clinical use which we can, with a very high degree of certainty, safely use to rule-out a heart attack.

The test also should allow people living in rural areas to get just as good care as in emergency departments because it could be deployed in rural hospital and general practices.  This would save many lengthy, worrying, and expensive trips for people to an urban emergency department.

This study was carried out by the Christchurch Emergency Department research group (director and senior author Dr Martin Than) in conjunction with the Christchurch Heart Institute (University of Otago Christchurch).  My colleague, Dr Joanna Young did much of the hard yards, and we thank our clinical research nurses and assistant for all they did to take blood samples, collect data, and lend a hand around the ED.  The manufacturer of the blood test, Abbott Point-of-care, provided the tests free of charge, but they were blinded to the results and all analysis was conducted independent of them.

How we envisage TnI-Nx may be used in the future to allow very early rule out of heart attacks

Please note – patients experiencing sudden onset chest-pain should always seek immediate medical attention.

I am fortunate to hold a Senior Research Fellowship in Acute Care sponsored by the Canterbury Medical Research Foundation, the Emergency Care Foundation, and the Canterbury District Health Board which enables me to participate in these studies.

ps.  You’ll have to read some of my older posts if you want to know why “Cheesecake files”