In today’s blog in PRIO’s blog series for Peer Review Week 2020, Marit Moe-Pryce, managing editor of the PRIO-owned and -run journal Security Dialogue, discusses the key role of editorial offices in mediating between authors and reviewers in the peer review process.
Photo: Joshua Ness
‘’The text suffers in every aspect that makes an academic publication an academic publication’’
‘’I think [the engagement with XXX in this draft] is a massive waste of everyone’s time. The notion that … is simply laughable and embarrassing’’
In March this year the Committee on Publication Ethics, COPE, initiated work on developing guidelines for if and if so, when, it is OK for editors to make changes to review reports. As a first step, COPE conducted a small (and not terribly scientific) survey of 145 individuals in editorial positions about their views on the topic. A report on the results was subsequently published and COPE promised to continue this work on developing guidelines. Most surprising, I found, was that as many as 15% said it was never OK to edit reports. Less surprising, perhaps, was that 25% said it was never OK to suppress a complete review, and that the most common reason for editing a reviewer report was ‘hostile or inflammatory language’.
So, while COPE works on its new guidelines, I wanted to share some impressions from an editorial office, where we are ‘the middleman’, situated between authors and editors, handling the review process. We see our role as protectors of the review process. We protect authors and reviewers alike, but we also protect the journal and its integrity. To do this well, intervention is sometimes necessary.
In this blogpost I want to explain why it is sometimes necessary to intervene, how it can be done, and also suggest why it may be that we sometimes receive surprisingly hostile reports. Finally, I intend to suggest that a good review process that protects and engages all parties could already go a long way in addressing much of the criticism commonly raised against double-blind reviews.
At the very heart of peer review is the intention to help authors understand the strengths and weaknesses of their argument so that they can improve and, eventually, publish the best possible version of that argument. ‘Eventually publish’ does not even necessarily mean publishing in the journal initiating the review process. Journals have different procedures and ambitions to support submitting authors in achieving their article’s full potential, some will be little more than facilitators, an inbox enabling the peer review process. Others will not only be ‘the middleman’ between the reviewers and the authors, but will also actively help authors navigate the feedback they receive and give suggestions for successful revision. In some instances, journal editors go even further and may act almost as mentors, to help for instance marginalized or early career scholars improve their chances of publication. Occasionally, this may even involve ‘altruistic’ reviews where, for instance, a young author is involved. The editors know the submission is unlikely to pass the review process, but that with the support of the review reports, they can help point the author in the right (alternative) direction. How the review process is organized very much depends on the journal, its priorities, care for its larger community, as well as the character of the submission in question. To ensure there is no bias in the review, however, it is important that the reviewers know as little as possible about any priorities or considerations already made by the editors prior to review. This is a real dilemma for the editorial office.
Moreover, it can also lead to a mismatch of expectations, where the reviewers do not know the editors’ plan for the submission and feel the journal is wasting their valuable time asking for a report. To be sure, many journals barely do any vetting before sending out for review, and it is then no surprise reviewers might occasionally react with hostility in frustration. While it should go without saying that a reviewer report should never be edited or suppressed because the editors disagree or dislike the substance of the report, there is a good reason why ‘never’ should not be the answer to COPE’s question. ‘Hostile or inflammatory language’ is unfortunately not uncommon and, based on my own experience, it seems to be on the increase.
On an annual basis the journal I work for receives two to three hostile reports that require our intervention. The two quotations with which this piece opens are actual quotations from review reports we have received – and subsequently edited. The intended recipient of the former review report was a junior scholar, the latter was a ‘star’ within the relevant field. (Lesson for junior scholars: anyone can receive fierce reviews, and for more advice on navigating the pitfalls of peer review, see Jørgen Jensehaugen’s piece in this blog series.) Both reviewers were excellent academics, well known and trusted by the journal – perhaps having a bad day on the job. (Lesson for us all: everyone can have a bad day on the job). Few, I think, would disagree that the tone in the given examples is inappropriate. Most will probably also understand how such feedback could be damaging to any scholar, but that a junior or marginalized scholar might be a lot more vulnerable both with regards to self-esteem and career progression. (For more on how peer review can adversely affect marginalized scholars, see Lynn Nygaard’s blog in this series.) It is hard to see how any author could get anything constructive out of a report that slays their argument with such hostility. It is, however, easy to see how someone might just want to give up, perhaps even on their academic career as such, as a result. Nonetheless, 15% of COPE’s respondents think we should have returned the un-edited version. But to what purpose, and at what cost?
It is, however, in my experience, never the reviewer’s intention to cause harm or be hostile. Their outbursts are made in the heat of the moment. (Lesson for all future reviewers: in the same way you should never send an email when you’re angry, you should also not submit your review report in anger). In all cases I have been involved with over the years, without exception, the reviewer has responded with an apology when I have contacted them after having read their report. In cooperation, and transparently, we have edited the reports to allow for a more appropriate tone. The reviewers’ points and criticisms are of course retained and although they will still usually be received as harsh, they are no longer hostile. Receiving harsh reports is unfortunately a part of the game and something most academics will have to expect at some point. (For more on how the problem of unprofessionally hostile reviews might be addressed, see Pavel Baev’s blog.) Had the reviewers known the context for the submission, or simply taken a deep breath before writing up their report, the report might have looked very different at the outset. In this sense, it is the journals’ role as the middleman and protector of the review process to also help the reviewers be the best version of themselves, including to intervene when necessary.
‘Hostile or inflammatory language’ is unfortunately not uncommon and, based on my own experience, it seems to be on the increase.
On the extremely rare occasions where the reviewer would refuse any edits to the tone, the journal’s responsibility has to be to protect the author. In these cases, it could also be defensible to suppress a review report as long as the valid points are communicated to the author through e.g. an editorial letter. The choice is not whether or not to share the valid criticism from a hostile report with the author, but just how to share it. The double-blind peer review process meets a lot of criticism, but rather than open reviews which comes with their own sets of issues, I am convinced that a more sustained hands-on engagement with the review process by the journals would be enough to address many of the concerns Sebastian Schutte discusses.
The key lesson I hope COPE, as well as young scholars at the receiving end of hostile reports, will be mindful of is that the reviewers rarely intend to be hostile. Reviewers, authors and editors, at the end of the day, have the same interest at heart. They all want to advance research and bring academia forward through good, quality-assured, scholarly research. What is then a little (transparent and careful) editing between friends?
Marit Moe-Pryce is Managing Editor of Security Dialogue.