In today’s instalment of PRIO’s blog series marking Peer Review Week 2020, Lynn P. Nygaard discusses ways in which peer review in its current form can reinforce existing inequities in the research system, and points to a need for more training in and reflection on the role of the reviewer to begin to address these concerns.
As researchers, we come across mountains of scholarly articles in our quest for knowledge, and most of us read outside our specific areas of expertise both thematically and methodologically.
How do we know whether we can trust what we are reading?
Apart from the critical thinking skills we have developed as scholars, the fact that the articles have been peer reviewed offers us some comfort: experts have looked at these articles and found them to be sound. In other words, peer review takes some of the burden off the reader by suggesting that the material we are viewing has passed some sort of quality control.
In contexts outside research, this gatekeeping might be even more important – for example, when we are reviewing the publications list of someone applying for a job or a grant, and their area of expertise is outside our own. It would be unreasonable to scrutinize every publication, partly because it would be excessively time consuming and partly because we might be ill-equipped to judge. (For a discussion of this issue, see Haakon Gjerløw’s blog in this series). So, we trust that the peer reviewers have done their job, turning away shoddy work at the gate.
Unfortunately, even within our areas of expertise, it is notoriously difficult to know excellence when we see it. Kravitz and colleagues, for example, find that the likelihood that two reviewers have the same assessment of a given article are “barely beyond chance.” And peer reviewers, just like casual readers, also fall prey to certain kinds of biases – such as “content-based bias, confirmation bias, bias due to conservatism, bias against interdisciplinary research, publication bias, and the bias of conflicts of interest.” These kinds of biases not only affect individual scholars, but also, more worryingly, might reinforce social inequalities.
Gender inequalities
It’s a familiar refrain in the productivity literature: men produce about 50% more scholarly publications than women; men are overrepresented among the most productive, and underrepresented among the least productive.
Perhaps not directly – partly because of the important role of blinding (when reviewers do not know who the authors are). This is meant to counteract the well-known Matthew effect, where the more prestige you accumulate as a scholar, the more you are likely to get (meaning that if you are famous enough, people will assume your work is brilliant even without reading it properly). When it comes to counteracting gender bias, the idea is that when the reviewer doesn’t know whether the author is male or female, then gender should not play a role in the reviewer’s assessment.
However, there is some indication that while bias might not operate against the author directly, it may affect assessments of the subject matter. Not only are inferences made about gender based on the topic, but topics that are perceived as “feminine” (anything related to women, gender, or children, for example) are considered less prestigious – less serious – than more “masculine” topics. In this case, it doesn’t matter whether the author is actually male or female; the bias will be against the subject matter. And we also know that issues related to women, gender, and children are largely researched by women. In this way, peer reviewers might inadvertently reinforce gender inequities in academia and create barriers for women in publishing their research.
Geolinguistic inequalities
English is often considered a lingua franca in academic publishing. Although English may not be the biggest first language in the world, it seems to be the biggest second language, and allows scholars from all over the world to communicate with one another.
While many good arguments can be made for how English has become a great equalizer – allowing scholars from many different countries to come together in a single discourse – critical voices have pointed out that it also means that anything not published in English is likely to be left out of the discourse altogether. Not only are non-English journals often left out of indexing services (such as Web of Science), but peer reviewers are known to request that authors substitute non-English references with something in English. While the argument given is often that the reviewer cannot properly evaluate a reference that they are unable to read, the effect is that authors writing in local languages will receive far fewer citations – even when their work is highly relevant to the discourse.
Effectively forcing scholars to write in English for an international audience means that scholars located outside the geolinguistic “center” (English-speaking countries) not only have to write in a second language or risk their professional development as scholars, but it also means that they constantly have to contextualize their work so that it can be understood by someone outside the Anglo point of reference. Indeed, a common complaint by scholars in the geolinguistic periphery or semi-periphery is that peer reviewers expect them to contextualize their work extensively – and justify how it might be possible to generalize from a peripheral case – while Anglo scholars are not forced to contextualize or justify in the same way.
Gatekeeping is an important task in academia, and gatekeepers have an important role. However, it can be difficult for anyone to step outside what they know, what they are familiar with. While it might be natural for peer reviewers to err on the side of caution – to be skeptical of topics and approaches they are not familiar with, as well as sources they are unable to read themselves – it also means that they might unconsciously reinforce social inequalities.
Few of us are taught how to review someone else’s work: we are expected to be able to give relevant feedback on someone else’s research simply on the basis of the research we have produced ourselves. But if we want to encourage greater diversity in the knowledge we create as scholars, and reduce social inequalities, perhaps we should treat peer reviewing as a separate skill that needs to be learned.
It we want to ensure that people are not turned away at the gate for no good reason, peer reviewers need well developed critical thinking skills – and perhaps a conscious sense of humility – not just a strong knowledge base.
Lynn P. Nygaard is Special Adviser on Project Development and Publications at PRIO, where she provides support in academic writing and professional development. Dr. Nygaard also conducts research on gender and identity in academic publishing and career progression.