2014年，剑桥大学计算机科学家尼尔·劳伦斯 被要求主持称为EuroNIPS（神经信息处理系统）的年度机器学习和计算神经科学会议。在会议记录是发布结果的常见方式这一领域中，提交给审稿的论文不到四分之一获得了演示位置，这是这类享有声望的会议的典型代表。 Google研究部副主席Corinna Cortes, suggested they take the opportunity to experiment with the peer review process and see what level of consistency it provides. Peer review is considered a ‘gold standard’ in science and generally receives widespread support from researchers, but there is evidence that it may not always be an impartial arbiter of quality. The EuroNIPS conference provided an opportunity to investigate this further.
Perhaps it is unreasonable to expect high levels of consistency from peer review. As Lawrence states, ‘you are sampling from three people who are not objective … they’ve got particular opinions’. This may not have once been a problem, but the current competitive nature of academia has made each peer reviewed decision highly significant. ‘If you have a funding rate of 5%, or 10%, you’re going to have very few winners and a lot of losers, and a lot of undeserving losers and some undeserving winners,’ says 约翰·波伦，来自美国印第安纳大学布卢明顿分校的复杂系统专家，他一直在研究分配资金的替代模型。
Aileen Day, Royal Society of Chemistry
Lawrence agrees that the root of the problem is how career-defining peer reviewed decisions can be. ‘Whether [a student] managed to get [their] paper [into the EuroNIPS conference] shouldn’t be the be-all-and-end all … unfortunately, it often is.’
对于赠款分配，还有一个问题，就是浪费时间浪费在写作和同行评审最终没有资金的建议上。根据澳大利亚研究人员的一项估计，他们准备了一年中总共提交的3,700项建议，代表了五个世纪的研究时间。2 But Bollen says this is not to criticise funding agencies. ‘A lot of really good people are involved … but a system that funds only 15% of the applicants, leaving 85% with zero money, cannot be efficient.’
同行评审的另一项指控是，评审员无疑会带有无意识的偏见。这些刻板的刻板印象是无意的，但根深蒂固并能够影响决策。 2019年，英国皇家化学学会（RSC）撰写了一份报告，探讨了2014-2018年向其期刊投稿成功中的性别差距。 RSC数据科学家Aileen Day发现在所有阶段（包括同行评审）都存在差异。3 ‘It was small, but it was significant,’ says Day. For example, while 23.9% of corresponding authors who submit papers are female, only 22.9% of papers accepted for publication have female corresponding authors. ‘The important thing is differences stack up through the whole publication process,’ explains Day.
The study also found that men and women behave differently as reviewers; ‘If you were a woman you were more likely to say major revisions, if you were a man, reject,’ says Day. Reviewers also give preferential treatment to their own gender.
Journals are clearly trying to ensure a better gender balance of reviewers and many institutions provide training for staff and reviewers to overcome unconscious biases , but there are differences in opinions on the effectiveness of these measures. A recent report by the professional HR body the Chartered Institute of Personnel and Development highlighted the ‘extremely limited’ evidence that training could change employee behaviour.
One idea for preventing bias is to hide the identity of the author of the paper or proposal, something the Engineering and Physical Sciences Research Council (EPSRC) have looked at. ‘We have trialled a number of novel approaches to peer review over the years including those involving anonymous or double-blind peer review,’ says head of business improvement Louise Tillman. But many reviewers say in relatively small academic communities it’s difficult to ensure anonymity .
在另一端，一些出版物已经进行公开的同行评审。例如，二月性质 announced it was offering the option of having reports from referees (who can still chose to stay anonymous) published alongside author responses . ‘In an ideal world, you might want the name of the reviewers to be open as well, but there’s a challenge there,’ says Lawrence, ‘[referees] might be unwilling to share a forthright opinion.’ The machine learning journal edited by Lawrence publishes peer reviews alongside papers. ‘That may have been my favourite innovation to come from the EuroNIPS experiment,’ he notes.
Some subjects seem to be slower to change than others. There are few examples of open review chemistry journals, for example, though there are signs of movement here recently – the RSC’s two newest journals offer authors the option of transparent peer review (publishing their paper’s peer review reports). Lawrence says the slow pace may relate to chemistry publishing being dominated by large society publishers. ‘Communities that have struggled are those that have professional institutions managing their reviewing,’ suggests Lawrence, who sees these types of organisation as slow to change.
If peer review is such a lottery, why not actually replace it with a lottery? Several funding bodies have trialled such systems. From 2013 the New Zealand Health Research Council has awarded its ‘explorer grants ’ worth NZ$150,000 (￡76,000) using a random number generator to select from all applications that were verified to meet its criteria . Several other funders have tested the idea: the Swiss National Science Foundation experimented with random selection in 2019 , drawing lots to select postdoctoral fellowships, and Germany’s Volkswagen Foundation has also used lotteries to allocate grants since 2017. Another, perhaps less brutal model, though not yet tested, suggests that applications not funded go back into the pot – creating a system more like premium bonds.4
Such systems ‘uniquely embody the worst of all possible worlds,’ says Bollen. ‘It’s essentially funding agencies and scientists saying we cannot do decent [peer] review… its almost spiteful.’
Bollen has come up with another idea for funding researchers: ‘I figured, why don’t we just write everybody a cheque?’ In 2019, inspired by mathematical models used in internet search engines, he published his idea for a ‘self-organised fund allocation’ system, where every scientist periodically receives an equal, unconditional amount of funding. The catch is that they must then anonymously donate a given fraction of this to other scientists who are not collaborators or from the same institution.6 Those scientists would then re-distribute a portion of what they receive. Bollen says the model ‘converges on a distribution and allocation of funding that, overall, reflects the preferences of everybody in that community collectively’ – perhaps the ultimate peer review. ‘The results could be just as good, just as fair as the systems that we have now, but without all of the overhead,’ he adds.
Tweaking the model could help solve current issues of bias, for example by mandating researchers to give a certain proportion of their funds to underrepresented groups. Of course such a system might favour academics who ‘talk a good game’ and disadvantage those in obscure fields, but that’s already the case in the present system, says Bollen. So far his model has had no takers, but has received lots of interest from colleagues and funding bodies. He is hopeful that after the current period of upheaval there may be an appetite for change.
Lawrence thinks we may just need to accept that peer review is always going to have flaws; ‘the idea that there’s a perfect, noise free system is the worst mistake.’ A number of recent well-publicised retractions, including a Lancet paper on Covid-19 hydroxychloroquine treatment, show that peer review is not error-proof. According to the website Retraction Watch at least 118 chemistry papers were retracted in 2019. Ultimately we may need to be realistic about what we mean by ‘gold standard’. ‘[Peer review] may be the best system we have for verifying research, but [that doesn’t always] mean [that] the research that’s rejected is somehow flawed or the research accepted is somehow brilliant,’ says Lawrence.
This article was updated on 26 August 2020. An earlier version referred to a comparison of publishing data from different countries, which had not been verified, and Aileen Day’s quoted use of the word ‘bias’ has been clarified to ‘differences’.
1 E L码头 等, PNAS, 2018, 115，2952（DOI：10.1073 / pnas.1714379115)
2 D L Herbert，A G Barnett和N Graves， 性质, 2013, 495314（DOI：10.1038 / 495314d)
3 A E Day，P Corbett和J Boyle， 化学科学, 2020, 11，2277（DOI：10.1039 / C9SC04090K)
4 F C Fang and A Casadevall, 生物技术, 2016, 7，e00422（DOI：10.1128 / mBio.00422-16)
500万刘 等, Res。整数同行评论, 2020, 5，3（DOI：10.1186 / s41073-019-0089-z)
6博伦 等, Ecol。 Soc。, 2019, 24，29（DOI：10.5751 / ES-11005-240329)