Grants go random to beat bias

Early results suggest the “partial” or “focal” randomisation method – in which the final selection from a potentially long list of worthy recipients is made by chance – has been effective in boosting diversity, improving grantseeker feedback and reducing the pain of assessment.

It’s a method also being used in several other countries: it’s been employed to determine who gets the Health Research Research Council of New Zealand’s Explorer Grants, the German Volkwagen Foundation’s Experiment! grants, the Swiss National Science Foundation’s grants (in the case of tie-breakers), and the Austrian Science Fund’s 1000 Ideas science seed funding.

British Academy and Leverhulme Small Research Grants is a fund that supports primary research in the humanities and social sciences, but assessors grapple with having to constantly knock back quality applicants for limited funds.

By far the British Academy’s most popular grants scheme in terms of applications and awards, the program has distributed funds to academics across nearly 100 institutions across the UK.

These awards are worth up to £10,000 (just over A$19,000) over two years, to cover the cost of research expenses. The Academy gets a significant portion of that funding from the Leverhulme Trust, one of the UK’s biggest grantmaking foundations.

Projects awarded funds as part of the scheme include a University of Birmingham project tracking the transition from school into employment in Ghana, and a professor’s investigation into how vulnerable adults manage their food insecurity.

The British Academy’s randomisation experiment was recently reviewed by the EDI Caucus, a group of UK researchers dedicated to greater equality, diversity and inclusion. A recent webinar featured the small grants program among other examples of ways of tackling bias in the assessment of funding applications.

Speaking to fellow grantmakers and researchers about the approach in a webinar, the head of research awards at the British Academy, Dr Ken Emond, said the scheme was extremely popular and most applications were worthy of funding. (View the recording on YouTube here.) Yet the available funds were not equal to the demand.

The organisation had partially randomised its grant allocations to reduce the load on assessors and increase fairness for applicants, Dr Emond said.

“Partial” randomisation means assessors first ensure applicants meet thresholds to be eligible for random selection. According to Dr Emond, 60% of applicants meet that measure.

“This is a scheme that attracts a lot of interest; we get many applications each year, most of which are fully suitable for funding. These are very small-scale, often experimental grants, commonly used as pilot studies for small-scale projects up to £10,000. It's the kind of grant where you can really take more of a risk because you're investing a small sum to enable interesting and exciting approaches to be tested out.”

Dr Emond told the webinar that many of the successful applicants went on to win much larger grants from other sources, and the funding also helped early-career researchers with their first opportunity to lead an investigation.

But in some ways, he said, the scheme was too popular: “The characteristic of the scheme is that we received many high-quality applications, far more than we can afford to fund.”

He said the downside of receiving so many applications was having to handle a large volume of assessments, all of which required careful examination.

“The time spent at the final stages of the process deciding very fine distinctions between who could be funded ­and who couldn't was significant,” he said.

He said the decision to trial randomisation had come from recommendations from an equality, diversity and inclusion (EDI) working group, with the aim of boosting the diversity of recipients across geographic regions, institutions and other characteristics.

Initially, the grantmaking team continued its normal peer reviews to ensure that applicants met the base criteria, including research quality, methodology and timeframes. Funding was then randomly allocated among the 60% of applications that met that criteria.

Dr Emond stressed there was no picking favourites at this point, or, as he described it, “pre-selection of the top candidates to be funded”.

He said randomisation made it much easier to give feedback to unsuccessful applications: grantmakers either alerted them to the fact that they were not as strong as other applicants (i.e. the 40%) or informed them they had not been picked in the random allocation.

After three rounds that employed this method, the small grants scheme had seen “a significant increase in the number of applications overall being submitted”, especially from diverse ethnic backgrounds. It had maintained gender diversity at roughly 50/50 but had seen a leap in the number of institutions represented in applications.

The small grants program’s funding rate is currently 27% of 900-odd applications per round, but the rate is set to rise following extra funding from government, the Leverhulme Trust and other funding partners, Dr Emond said.

Dr Emond said the trial would continue for at least three more rounds during the evaluation period, but “the overall impression is [that it is] very successful in raising interest among a wide range of people who perhaps self-disqualified themselves in the past.”

More information

British Academy/Leverhulme Small Research Grants

EDI Caucus research: Peer review bias in the funding process