As misinformation and disinformation run rampant, questioning the utility of peer review is sure to raise eyebrows, blood pressure and tempers among academics and health-care professionals alike.
Faith in institutions defined by their commitment to empiricism is waning, and I can understand the desire to look to peer review as a guardian of scientific rigor. However, I fear peer review may be a troubled guardian of a gate that has already been breached.
Peer review is a process that calls upon people with deep expertise to review materials within their field to determine if emerging knowledge or methods are valid and worthy of publication and/or funding. Peer review was institutionalized by the Royal Society of London in the 18th century when publication and dissemination of new knowledge through printed publication was time and resource intensive. As not all new scholarly findings could be shared, only the most compelling and impactful works could be afforded publication. It was not only practical to have a process to identify top materials for publication, but it was also helpful in building institutional and authorship reputations; being a reviewer or author of the Royal Society was highly distinguishing.
Over time, it became a common belief that the peer review process ensured quality in the publication or scientific process. Submitting a paper or study protocol for review has permitted experts outside a research team to identify errors in text or process and provide feedback to the authors, thus promoting iterative refinement of their work.
In addition to the resource stewardship and quality control arguments, peer review is psychologically appealing. Humans are social beings that crave endorsement and acceptance in their peer groups, particularly if group members are viewed as wise or powerful.
So, it is not surprising that peer review became so popular with little challenge (or actual evidence that it works).
Today, the sharing and consuming of information is cheap and easy but the resource intensity of peer review is extraordinary. Though there is no need for an editorial board, or even a journal, to share new knowledge, it was estimated that the monetary value of time American reviewers alone spent in peer review activities in 2020 was $1.5 billion USD. Yet, we continue to use journals and peer reviewed venues to vet emerging knowledge even though the opportunity costs paid by the scientific community are absurd.
The evidence would suggest that peer review is an imperfect process at best. Peer review is effective at limiting access to publication/funding but may not work as well as we would like to think when it comes to identifying valid or impactful science.
There are several studies that have demonstrated that ratings across peer reviewers are neither consistent nor reliable.
There are several studies that have demonstrated that ratings across peer reviewers are neither consistent nor reliable. A review of the now retired Foundations Grant Program at the Canadian Institutes of Health Research illustrated that peer reviewers are influenced by personal factors (their own or the applicant’s) and revealed systemic gender bias in their peer review process. There are well established “trends” in publication suggesting that manuscripts examining “hot” topics may be more favourably reviewed than others. There is also a well-accepted bias against submissions that are non-English and from low- and middle-income countries. This bias is so normalized that we explicitly test for publication bias in systematic review methodology.
Recognition of the systemic exclusion of diverse and Indigenous scholars has increased our awareness that the knowledge we have about our health and medicine has been shaped by a select few and permitted the establishment of practices that perpetuate and promote health inequity. But it should be no surprise that peer review would contribute to epistemological bias, as at its core, peer review is a process designed to exclude people from contributing to scholarly discourse.
In short, our long tradition of peer review has not protected the knowledge-building process from human failings or biases. And in its effort to gate-keep, may have contributed to harm by not valuing diverse thought or highly novel topics or approaches.
Peer review, in its current form, does not appear to be serving its intended purpose. While there is active exploration of innovative approaches to peer review to ensure the practice is effective, transparent and equity promoting, many innovations are just spins on what we have been doing for the last two centuries (and are being proposed by an academy and publishing industry that have peer review as an institutional cornerstone).
We live in a time when ideology-driven political leaders actively feed conspiratorial narratives about health and medicine. Ensuring that scientifically robust information is identified and valued has never been more important; promoting interdisciplinary/intersectoral dialogue about the science of health is critical.
There is little doubt that we are descending into a new dark age of disinformation, and we need to think seriously about how to spark a new enlightenment. Somehow, I doubt that Reviewer # 2 will bring the light we need.
