Tag Archives: Performance Based Research Fund

PBRF: The end is nigh

I’d like to say the end is nigh for the performance-based research fund (PBRF), full stop. A few months ago, I demonstrated how the expensive and tedious production of evidence portfolios by 7000 academic staff will do nothing to change the redistribution of research funding – the purported reason for PBRF. So, I’d like to say the end is nigh because the minister responsible (Hon. Chris Hipkins) has seen the light and pulled the plug. But, alas, it is simply that all portfolios have now been submitted and so await assessment by the peer review panels . About 250 people serve on these panels, nearly all of whom are Professors, most from New Zealand but a sprinkling from Australia and elsewhere.  They represent the gathering of some of the best minds in the country.  From my perspective it is a terrible waste  of time for them and of tax-payers’ money for the rest of us. 

In completing my portfolio I received a message concerning citation counts that “Panels are not a fan of Google scholar as they think the counts are over-inflated. You can use this but also supply cite counts from either Scopus or WoS.” Frankly, I think the panellists are far too intelligent to worry about this and I expect that they realise that while Google scholar counts are over-inflated, that Scopus (owned by Elsevier!) and WoS under-count (eg by not counting book chapters, leaving out some journals etc).  What matters, if citations have to be used at all, is that apples are compared with apples.  I’ve discussed some of these problems recently.  Before I suggest a solution that doesn’t require 250 Professors sitting in days of meetings, or 7000 academics spending days in completing evidence portfolios, I’ve produced a graphic to illustrate the problem of comparing apples with oranges.  Google scholar ranks journal according to the 5-year h-index. These can be explored according to the various categories and sub-categories Google Scholar uses (here). Visually each of the 8 major categories has different numbers of citations and so of the h-indices.  For example, Social Sciences is a small fraction of Health and Medial Sciences, but is larger than the Humanities, Literature & Arts.   Within each category there are large differences between sub-categories.  For example, in the Health & Medical Sciences category a cardiologist publishing in cardiology journals will be publishing in journals where the top 20 h-indices range from 176 to 56.   However, the Nursing academic will be publishing in journals whose top 20 h-indices range from 59 to 31.  So what is needed is a system that takes into account where the academic is publishing.

Visualisation of Google Scholar’s h-5 index Categories (large ellipses at the bottom) and sub-categories (smaller ellipses). Each sub-category ellipse represents in height and area the sum of the h-indices for 20 journals within that sub-category.

Google Scholar, which, unlike WoS and Scopus, is open and public, can be scraped by just three lines of code in R (a free and open programming language) to extract the last 6 years of published article and their citations for any academic with a profile on Google Scholar.  Thousands of NZ academics already have one.  Here’s the code which extracts my last 6 years of data:

pubs<-get_publications("Ig74otYAAAAJ&hl") %>% filter(year>=2012 & year <=2017)


The “Ig74otYAAAAJ&hl” is simply the unique identifier for me which is found in the URL of my Google Scholar profile (https://scholar.google.co.nz/citations?hl=en&user=Ig74otYAAAAJ&hl).

I’ve also been able to scrape the list of top 20 journals and their h-index data for the 260 sub-categories from Google Scholar.  Here is what Cardiology looks like:

Google Scholar’s tops 20 journals for Cardiology as at 13 July 2018: https://scholar.google.co.nz/citations?view_op=top_venues&hl=en&vq=med_cardiology

So, how do we use all this data to compare academics without them having to submit screeds of data themselves?  All that needs is for them to be registered with their Google Scholar identity and for there to be an appropriate formula for comparing academics.  Such a formula is likely to have several components:

  1. Points for ranking within a category. For example, 20 pts for a publication ranked first in a subcategory, down to 1 pt for a publication ranked 20th and, say, 0.5 pts for ones not ranked.
  2. Points that reflect the number of citations a paper has received relative to the h-index for that journal and with a factor that accounts for the age of the paper (because papers published earlier are likely to be cited more).  For example, #citations/Journals 5y h-index * 2/age[y] * 20.  I use 20 just to make it have some similar value to that of the ranking in point 1 above.
  3. Points that reflect the author’s contribution.  Perhaps 20 for first author, 16 second, 12, 8, and 4 for the rest + a bonus 4 for being Senior author at the end.

Here’s a couple examples of mine from the last 6 years:

Pickering JW, Endre ZH. New Metrics for Assessing Diagnostic Potential of Candidate Biomarkers. Clinical Journal Americac Society Nephrology (CJASN) 2012;7:1355–64. Citations 101.

The appropriate sub-category is “Urology & Nephrology” (though I wonder why these are grouped together, I’ve published in many Nephrology, but never a Urology journal).

  1. Ranking:  12 points.    [CJASN is ranked 8th, so 20-8 = 12]
  2. Citations:  10.8 points. [ CJASN 5y h-index is 62. Paper is 6 years old. 101/62 * 2/6 * 20 =10.8]
  3. Author: 20 points [ 1st author]
  4. TOTAL: 42.8

Similarly for:

Flaws D, Than MP, Scheuermeyer FX, … Pickering JW, Cullen, L. External validation of the emergency department assessment of chest pain score accelerated diagnostic pathway (EDACS-ADP). Emerg Med J (EMJ) 2016;33(9):618–25. Citations 10.

The appropriate sub-category is “Emergency Medicine”  (though I wonder why these are grouped together, I’ve published in many Nephrology, but never a Urology journal).

  1. Ranking:  12 points.    [EMJ is ranked 8th, so 20-8 = 12]
  2. Citations:  10.8 points. [ EMJ 5y h-index is 36. Paper is 2 years old. 10/36 * 2/2 * 20 =5.6]
  3. Author: 4 points [ I’m not in the top 4 authors or senior author]
  4. TOTAL: 26.8 pts

This exercise for every academic could be done by one person with some coding skills.  I’m sure it could be calibrated to previous results and funding allocations by taking citations and papers for an earlier period. There may need to be tweaks to account for other kinds of academic outputs than just journal articles, but there are plenty of metrics available.

To summarise, I have just saved the country many millions of dollars and allowed academics to devote their time to what really matters.  All it needs now is for the decision makers to open their eyes and see the possibilities.

(ps. even easier would be to use the research component of the Times Higher Education World University Rankings and be done with it).


More on the PBRFs new clothes

A few of weeks ago I outed the multi-million-dollar exercise that is the Quality Evaluation component of the performance based research fund (PBRF) as a futile exercise because there was no net gain in research dollars for the NZ academic community.  Having revealed the Emperor’s new clothes, I awaited the call from the Minister in charge to tell me they’d cancelled the round out of futility.  When that didn’t come, I pinned my hope on a revolt by the University Vice-Chancellors. Alas, the VCs aren’t revolting.  This week, my goal is for there to be mass resignations from the 30 or so committees charged with assessing the evidence portfolios of individual academics and for individual academics to make last minute changes to their portfolios so as to maintain academic integrity.

I love academic metrics – these ways and means of assessing the relative worth of an individual’s contribution to academia or of the individual impact of a piece of scholarly work are fun.  Some are simple, merely the counting of citations to a particular journal article or book chapter, others are more complex such as the various forms of the h-index. It is fun to watch the number of a citations of an article gradually creep up and to think “someone thinks what I wrote worth taking notice of”.  However, these metrics are largely nonsense and should never be used to compare academics.  Yet, for PBRF and promotions we are encouraged to talk of citations and other such metrics.  Maybe, and only maybe, that’s OK if we are comparing how well we are performing this year against a previous year, but it is not OK if we are comparing one academic against another.  I’ve recently published in both emergency medicine journals and cardiology journals.  The emergency medicine field is a small fraction the size of cardiology, and, consequently, there are fewer journals and fewer citations.  It would be nonsense to compare citation rates for an emergency medicine academic with that of a cardiology academic.

If the metrics around individual scholars are nonsense, those purporting to assess the relative importance (“rank”) of an academic journal are total $%^!!!!.  The most common is the Impact Factor, but there are others like the 5-year H-index for a journal.  To promote them, or use them, is to chip away at academic integrity.  Much has been written elsewhere about impact factors.  They are simply an average of a skewed distribution.  I do not allow students to report data in this way.  Several Nobel prize winners have spoken against them.  Yet, we are encouraged to let the assessing committees know how journals rank.

Even if the citation metrics and impact factors were not dodgy, then there is still a huge problem that faces the assessing committee, and that is they are called on to compare apples with oranges.  Not all metrics are created equal.  Research Gate, Google Scholar, Scopus and Web of Science all count citations and report h-indices.  No two are the same.  A cursory glance at some of my own papers sees a more than 20% variation in counts between them.  I’ve even paper with citation counts of 37, 42, 0 and 0.  Some journals are included, some are not depending on how each company has set up their algorithms. Book chapters are not included by some, but are by others. There are also multiple sites for ranking journals using differing metrics.  Expecting assessing committees to work with multiple metrics which all mean something different is like expecting engineers to build a rocket but not to allow them to use a standard metre rule.

To sum up, PBRF Evidence Bases portfolio assessment is a waste of resources, and encourages use of integrity busting metrics that should not be used to rank individual academic impact.

Performance Based Research Fund: a net zero sum game

Throughout the land more than 7000 academics are awake night after night and suffering.  They are scrambling to gather evidence of just how great they have performed over the last six years. A conscientious bunch, they perform this task with their usual attention to detail and desire to impress (I didn’t say they were modest!).  Ostensibly, this exercise is so that their institutions can get a greater piece of the Government research fund pie – the Performance Based Research Fund (PBRF).  According to the Tertiary Education Commission PBRF is “a performance-based funding system to encourage excellent research in New Zealand’s degree-granting organisations.”  It may well do that, but, I contend, only by deception.

In what follows I am only concerned with the Quality Evaluation part of PBRF – that’s the bit that is related to the quality of the Evidence Portfolio (EP) provided by each academic. The data is all taken from the reports published after each funding round (available on the TEC website).

In 2012 the total funding allocated on the basis of EPs was $157 million with nearly 97% of it allocated to the country’s 8 universities.  This total amount is set by Government fiat and, here is the important point, in no way depends on the quality of the Evidence Portfolios provided by those 7000+ academic staff.   In other words, from a funding perspective, the PBRF Quality Evaluation round is a net zero sum game.

PBRF Quality Evaluation is really a competition between degree granting institutions.  I find this strange given the Government has been trying to encourage collaboration between institutions through funding of National Science Challenges, nevertheless a competition it is.

In the table we see the results of the Quality Evaluation for the previous three funding rounds ( 2003, 2006 and 2012).  Not surprisingly, the larger universities get a larger slice of the pie.  The pie is divvied up according to a formula that is based on a weighting for each academic according to how their research has been evaluated (basically A, B or C), multiplied by a weighting according to their research area (eg law and arts are weighted lower than most sciences, and engineering and medicine are weighted the highest), multiplied by the full time equivalent status of the academic.   In theory, therefore, an institution may influence their proportion of funding by (1) employing more academics – but this costs more money of course, so may be defeating, (2) increasing the proportions of academics in the higher weighted disciplines (some may argue this is happening), and (3) increase the numbers of staff with the higher grades.  I will leave it to others to comment on (1) or (2) if there is evidence for them.  However (3) is the apparent focus of all the activity I hear about at my institution.   There are multiple emails and calls to attend seminars, update publication lists, and to begin preparing an Evidence Portfolio.  Indeed, in my university we had a “dry run” a couple of years ago, and it is all happening again.

Now, I come to the bit where I probably need an economist (it is my hope that this post may influence one to take up this matter more).  Because it is a net-zero sum game, what matters is a cost-benefit analysis for individual institutions.  That is, what does it cost the institutions to gather EPs compared to what financial gain is there from the PBRF Quality Evaluation fund?  If we look at the 20012-2006 column we see the change in percentage for each institution.  The University of Auckland for example increased its share of the pie by 1.3% of the pie.  This equates to a little under $2M a year.  As the evaluations happen only every 6 years we may say that Auckland gained nearly $12M.  What was the cost? How many staff for how long were involved?   As there are nearly 2000 staff submitting EPs from Auckland another way of looking at this is that the net effect of the 2012 Quality Evaluation round was a gain of less than $6000 per academic staff member over 6 years.  How much less is unknown.

The University of Otago had a loss in 2012 compared with 2006.  Was this because it performed worse – not at all, indeed Otago increased how many staff and the proportion of staff that were in the “A” category and in the “B” category. This suggests improved, not worsened, performance.  I think that Otago’s loss was simply due to the net zero sum game.

Much more could be said and questions asked about the Quality Evaluation, such as what is the cost of the over 300 assessors of the more than 7000 EPs?  Or perhaps I could go on about the terrible use of metrics we are being encouraged to use as evidence of the importance of the papers we’ve published.  But, I will spare you that rant, and leave my fellow academics with the thought – you have been deceived, PBRF Evidence portfolios are an inefficient and costly exercise which will make little to no difference to your institution. 

Spin Doctors go to work on PBRF

University of Taihape:  Doctor Doctor I’ve got a 1.7 on my PBRF

Doctor Spin: Never mind son, your Gumbootlogy results make you the healthiest tertiary education provider in the country.  Let’s talk about that, shall we?

Scoop.co.nz has all the spin from the universities and polytechnics this morning as they try and give the impression that they are the best.  At times like this I am ashamed to be an academic.  One of the worst of sins is to cherry pick data to make your self look good.  We are used to this from certain sectors of society, but we should expect better from our educational institutions. Unfortunately, the culture of self promotion above all else has taken hold in our hallowed halls.

For those who are unaware of what I am talking about, around 18 months or so ago all academics in the country had to put forward a “portfolio” to demonstrate just how research active they were.  This is the Performance Based Research Fund (PBRF) exercise held every 6 or so years. Groups of academics under government oversight then went about scoring each academic in a process that has taken 15 months.  The result is that every academic has been given a grade A , B, C or not research active.  The grades of academics in each institution are then thrown into four different formula – each has additional information about different aspects of the institution (eg numbers of postgrad students).  Four numbers result.  This gives Doctor Spin plenty to play with. The numbers are also what are used to allocate hundreds of millions of dollars of research funds – here in lies the importance of PBRF to the institutions. A number is also provided for each of the self selected academic units that the institutions provided to the Tertiary Eduction Commission.  If they don’t score well in any of the four overall grades (comparative to other institutions their own size), then they can pick a favourable number from one of their academic units and talk about that. More grist for the spin mill.

Academics are notoriously competitive – obviously a good trait when it drives them to produce better and better research. I certainly have some of that streak in me. However, it is not  helpful when it results in attempting to pull the wool over the eyes of the public as happened yesterday.  The PBRF is a complex system designed to find a way to allocate research funds and hopefully improve research quality.  Academics will argue until the cows come home if it does this fairly. It certainly is a very expensive exercise. It certainly focusses institutions on the importance of research, which is a good thing.  Remember, the teaching in our universities (not polytechnics) is required by law to derive from research.  However, as a small country where the combined size of all our universities is only that of some of the larger overseas universities I wonder if such a inter – institution competitive model is the best for the country?  Perhaps the story should be an evaluation of cost benefits of the exercise. Is this the best method of allocating funds? Such a story should also consider if the competition is enhancing or detracting from the quality of research – after all in almost any field experts are spread across institutions.  Collegiality is a major driver of good research – does PBRF hinder that?


If you want to check out the PBRF results in detail your self you can download a PDF from the Tertiary Education Commission here.

Disclaimer:  If you think my skepticism about PBRF is sour grapes because of a “poor” grade, then you’d be wrong.