Reviewed Peer Review

In today’s blog in PRIO’s series marking Peer Review Week 2020, Sebastian Schutte discusses some of the weaknesses of the current blind peer review system and points to a possible solution: reviewing peer reviews.

Recent debates on how to improve scientific publishing have largely focused on open access. This is understandable, but access to the finished product is only one aspect of publishing worth improving.

Currently, peer review is mostly done in a closed fashion: editors pick reviewers they deem knowledgeable in a certain area. Those reviewers then get to accept or decline reviewing a manuscript. As the incentives for writing reviews are very small, many invitations to reviewers are rejected. But the problems do not end there.

The clear intention here is to put the scholarly work in the foreground and prevent biases, reciprocity, or personal sentiments from informing the reviews. However, the reality is oftentimes very different. Reviewers can simply look for working papers with identical titles online and acquire institutional affiliations of authors. This in effect, makes the double-blind review process single-blind, bringing back all those biases and sentiments that double-blind reviews are supposed to eliminate.

Most problematically, the reviewers deemed knowledgeable by editors tend to be competitors of the authors, oftentimes working on equally timely topics, using similar methodology, at comparable levels of academic seniority. What could possible go wrong here? According to the people over at Shit My Reviewers Say, who scream into an internet version of their pillows, this setup is prone to sabotaging the publications of competitors. If anything might look too good on the CV of a competitor, why not focus on minute problems and recommended rejection of manuscript?

A single review is a powerful weapon: as acceptance rates in renowned journals are oftentimes below 10% and editorial decision can take 6-9 months, obstructive reviews are a great way to weaken competitors in the run-up for jobs, grants, and fellowships. The ensuing rage can be sensed in the comment section of Political Science Rumors.

Needless to say, single-blind abuse of power also hampers scientific progress. Especially manuscripts breaking new ground threaten work that is still unpublished or in preparation and such contributions tend to receive petty pushback, effectively delaying their publication. And when reviewers raise valid points, their reviews generally do not get published, leaving the task of reassessing the merits of publications to future audiences.

A remedy has been proposed: open peer review. Instead of keeping identities secret, reviewers and authors should know whom they are dealing with. Proponents of this approach claim that civility and reason in the review process can be restored when identities and reputations can be tied to reviews. In response to this approach, several concerns have been raised: open peer review can simply reinstate the problems that blind peer review was supposed to solve. Biases and personal sentiments can seep into reviews.

Reciprocity, patronage, and nepotism can interfere with the scientific assessment of scholarly work.

Additionally, reviewers might be more reluctant to accept reviewer tasks. As stated above, the incentives to review are generally very low and when one adds the costs of potentially crossing peers that might retaliate later on, increased reluctance to review might be the result. Versions of these pros and cons have been discussed extensively (see, for example this discussion in Nature Neuroscience), and experiments with open peer review suggest minor overall benefits.

However, a third option is imaginable, which addresses several issues of blind review systems while also generating incentives for reviewers: reviewed peer review.

Reviewed peer review simply asks one additional task of reviewers: briefly comment on and score the reviews provided by other reviewers. The task is manageable, given that all reviewers will have read any manuscript in detail and have written up their own detailed response. This step generates accountability for those that misuse their powers as a way to simply harm potential competitors. Additionally, the reviews and mutual scores can be published so that future audiences are presented not only with final articles, but also with an initial sense of how unified the reviews were in their assessments.

When reviews are conducted with real names, reviewers can build up reputations for the significant work they put into advancing scholarship. The qualities they put on display here are not the same as the ones that will get them published themselves. Rather, they can demonstrate their abilities to deeply read forthcoming work and intelligently comment on, improve, and correct inevitable faults. For hiring decisions, and for judging abilities in supervision and mentoring, these qualities are central.

Even if real names are removed from the equation, reviewed peer review generates accountability.

Editors will get a measure of which reviewers can be trusted to raise valid points and publishing anonymous mutual ratings still conveys precious information to future audiences.


Sebastian Schutte is a Senior Researcher at PRIO. His research investigates political violence, insurgencies, and civil wars using quantitative methods.

Share this: