Tag Archives: National Science Challenges

What the HRC should have done

The system is broke.  It is no better than a lottery.  The Health Research Council tacitly acknowledged this last year when they introduced a lottery to their grant funding round.  The lottery was for three grants of $150,000 each.  These “Explorer Grants” are available again this year.  The process went thus: HRC announced the grant and requested proposals;  proposals were required to meet simple requirements of transformative, innovative, exploratory or unconventional, and have potential for major impact;  proposals were examined by committees of senior scientists;  all that met the criteria were put in a hat and three winners were drawn out.

116 grants were received, 3 were awarded (2.6%!!!). There were several committees of 4-5 senior scientists. Each committee assessed up to 30 grants.  I’m told it was a couple of days work for each scientist. I’m also told that, not surprisingly given we’ve a damned good science workforce, most proposals met the criteria. WHAT A COLOSSAL WASTE OF TIME AND RESOURCES.

Here is what should have happened:  All proposals should have gone immediately into the hat.  Three should have been drawn out.  Each of these three should have been assessed by a couple of scientists to make sure they meet the criteria.  If not, another should be drawn and assessed.  This would take about a 10th of the time and would enable results to be announced months earlier.

Given that the HRC Project grants have only about a 7% success rate and that the experience of reviewers is that the vast majority of applications are worthy of funding  I think a similar process of randomly drawing and then reviewing would be much more efficient and no less fair.  Indeed, here is the basis of a randomised controlled trial which I may well put as a project proposal to the HRC.

Null Hypothesis:  Projects assessed after random selection perform no differently to those assessed using the current methodology.

Method:  Randomly divide all incoming project applications into two groups. Group 1: Current assessment methodology.  Group 2: Random assessment methodology.  Group 1: assess as per normal aiming to assign half the allocated budget.  Group 2: Randomly draw 7% of the Group 2 applicants;  assess;  draw more to cover any which fail to meet fundability (only) criteria;  fund all which meet this criteria in order they were drawn until half the allocated budget is used.

Outcome measures:  I need to do a power calculation and think about the most appropriate measure, but this could be either a blinded assessment of final reports or a metric like difference in numbers of publications.

Let’s hope that lessons are learnt when it comes to the processes used to allocate National Science Challenges funds.

Totally Underwhelming

What do you get when you cross dozens of New Zealand’s best scientists with a myriad of Ministry officials?  The answer is the underwhelming reports from the 10 June workshops of the National Science Challenges. If every there was need for proof that science by committee does not work, here it is.  Each report consists of a series of power point slides with dot points. About 3 slides for each challenge pertain to confirming or changing the pre-workshop goals/themes (yes there were pre-workshop meetings in May to get these together), and then about 4 or 5 slides on “Indicative research programs.”  These handful of slides were the output of on average 44 people per group consisting of scientists, industry or other “user group” representatives, and ministry representatives.  The people I know who attended a workshop were senior and very very competent people.  The problem is not the people, but the process.  I saw nothing in the reports to inspire, and nothing that couldn’t have been cobbled together by one person after receiving emails with suggestions.  Most of the “indicative research programs” were simply restatements of the obvious questions in the field.  There was no meat. There was only one concrete proposal (High Value Nutrition proposed establishing a “Biomarker Centre”).

Recall that the 10 challenges will have $133.5M to spend over 4 years, about $3.34M per challenge per year.  The June meeting, I estimate, just cost about $0.5M in salaries for the day, overheads on those salaries, travel etc.  For $0.5M we have been given about 60 power point slides most of which could have been reproduced in half an hour or so by one or two of the scientists from each group – “Mr Speaker, would the Minister of MBIE please explain why one power point slide costs $8,300?”  Indeed, I have no doubt if the $3.34M was used to support half a dozen scientists and they were told – here’s the field (name of the challenge), you have $3.34M for each of the next 4 years, do some good science for the country in this field, – then it would be done.  Furthermore, the outcome would be at least, if not more, valuable than any multitude of small projects that are likely to emerge from the June workshops (but only after much more time and $ has been spent on more meetings, development of requests for proposals and a grant funding process that will take up many more millions and waste time for 90+% of the applicants; much as what happens now with other government funding models).  The Great Science Challenge for New Zealand is not how to define the projects, but how to provide long term sustainable funds for scientists who already know the projects.

NZ Scientists already well engaged in the National Science Challenges

Sorry to post twice in a day, but this is worth a look.  In the April Cabinet paper (here as a pdf) is an interesting graph which shows that 72.5% of current Vote Science and Innovation contestable and CRI core funding already goes to the newly announced National Science Challenges.  By my calcs the additional funding will push this to 78% assuming no increase in the “Research outside the Challenges” slice of the pie.  Have the challenges, therefore, merely reflected what is going on already in the science community?  One of the unanswered questions is “Which challenges will get the most funding from the new money?”  The graph shows the status quo.

The tao of science missed by National Science Challenges

The challenges are out. The committee has spoken. And now the critics respond.  Word on science street and in the media goes a bit like this:

Brilliant $73M more for science in New Zealand.  Well done Steven Joyce and the National Party.

Lacking in lustre.  These challenges are all a bit predictable. [eg Prof Hendy here]

Damn.  My research does not fit any of the challenges. [eg Dr Wiles here]

I sympathise with each of these opinions.  The National party has set a goal of 0.8 percent of GDP for science.  This is to be applauded. They have chosen a path of narrowing the scope of science to ensure it meets their own ideology of “government’s job is to grow the economy”.  This is reflected in the challenges and the language around them.  For example the challenge “High value nutrition: research to develop high value foods with health benefits” in the Peak Report document states:

There is enormous capacity to leverage both our primary industry and medical research to discover, validate and develop nutritional products with proven health benefits of significant market potential.

Some scientists seem to think that economic goals some how “devalue” science.  I am rather more pragmatic in suggesting that an economic return is an inevitable result of doing science.  The difficulty, though, is that any attempt to pick winners – and that is what the National Science Challenges does, fails to recognise that science at its best is not shackled but free to explore and expand.  Science by its very nature is at a frontier and a journey into lands unknown.  A pathway cannot be chosen for it and any attempt to do so will as often as not go straight past the pot of gold.

The National Science Challenges have been chosen by committee – there are “winners” and “losers” and the result is necessarily bland.  This is inevitable when science is done by committee.  Great science comes from great scientists who are driven to great discoveries.  It is driven by leadership, and leadership never comes from a committee.  On Morning Report this morning the interviewer and Prof Hendy both mentioned the US Space Program as an example of a truly exciting and great science challenge.  That challenge came from a great leader, President Kennedy, and while driven politically, the political goal was the same as the science vision.  Sadly, once the political goal had been reached the politicians turned elsewhere and the science community was left holding on to a few rocks and a vision shattered.

From my perspective what is needed for science in this country even more than challenges is vision and visionaries.  We need to fund scientists first and projects second.  Sadly, we have that priority completely around the wrong way.  Dr Wiles who ironically was one of the faces of science on the television campaign encouraging public submissions on the challenges is disappointed that her area of research, infectious diseases, is not acknowledged in a challenge.  I am disappointed that enthusiastic talented scientists like Dr Wiles are not directly receiving 3, 5, 10 year’s of salary and research cost support from this new money to pursue their vision. It’s not so much the topic of research as the researcher that counts.  I have a challenge for the New Zealand government.  And that is for their science policy to be evidence based (see Grant Jacobs’ blog post).  Part of that puzzle is whether it is best to fund researchers or to fund projects. This is why I say the Challenges have missed the tao of science – they are not in harmony with the way science is really done.  Let us run a trial.  Randomly select ten scientists and fund their salaries and $100K a year and let them pursue whatever they want.  Compare this to the results of randomly selected National Science Challenge funded projects with the same number of scientists involved.  The title of the trial could be “Is picking winners better than letting winners pick?