Peer Review and Societal Impact of COVID-19 Research

In this final instalment of our blog series marking this year’s Peer Review Week, Senior Research Nicholas Marsh looks at how Reddit users are accessing research on the ongoing COVID-19 pandemic and finds that non-peer-reviewed publications pose a real challenge to peer-reviewed publications as a way to disseminate research to the general public. This points towards important debates to be had in the years ahead on the role of peer review in the research lifecycle.

Photo: El G via Unsplash

Research has been at the heart of global attempts to address and understand the COVID-19 pandemic. This blog post examines the importance for achieving societal impact of different forms of publication concerning COVID-19 research. It has often been claimed that peer-reviewed articles in academic journals are the ‘gold standard’ of research publication, but for the wider readership in society, have they been most significant in the fast moving global crisis caused by the pandemic?

Assessing societal impact is not easy. Academic researchers commonly use citations and journal rankings as a means to measure the impact of their work among other scholars. However, researchers and bodies that fund their work also wish to understand the wider societal impact and relevance of their work. Of the circa 54 000 academic articles that have already been published on COVID-19, which are most useful in helping people understand the pandemic and attempts to ameliorate the disease? Unfortunately there is no easily available way to discover what research is actually being read by people outside the academy, and which publications have the most societal impact.

A fast-moving crisis like the current pandemic may provide incentives for other forms of publication that circumvent the time-consuming process of double-blind peer review. In particular, as an alternative to the traditional priority given to peer-reviewed articles in academic journals, in recent years some researchers have instead disseminated research using ‘preprints’ which are uploaded to a site such as bioRxiv.org without first having been through the double-blind review process. Preprints have the advantage that they can be published quickly and can then be openly discussed by a wide range of scholars and laypeople (which is often referred to as post-publication review). (For a further discussion of post-publication review, see Sebastian Schutte’s blog in this series.) Research published as a preprint may later be revised and submitted to an academic journal, or the preprint could be a stand-alone publication.

In order to understand the societal impact of different forms of publication I’m using Reddit, which is a community of some 300 million users who discuss all manner of subjects in over 100,000 active forums. One of those forums is called COVID-19 and has over 300,000 members. Its purpose is to discuss scientific research related to the pandemic and its moderators ensure that users only start discussions related to ‘scientific sources’, which include ‘peer-reviewed research, pre-prints from established servers, and information reported by governments and other reputable agencies’. Secondary sources such as news articles are to be discussed elsewhere. The COVID-19 forum is a place where a large number of interested lay people can find and discuss the latest research.

For the purpose of this blog, Reddit has a useful feature in that users can upvote or downvote posts about a specific publication. Those votes are then used by Reddit to produce a ranking. The score produced by the number of votes is a form of Altmetric, which I’m using to assess which research has the most societal impact. For example, the most highly voted publication is a report on the use of masks, which received a score of over 18,000 (meaning that at least that many people had upvoted it). I used the score to identify the top 100 publications that had been discussed on the COVID-19 forum.

I identified eight different types of publication, something which underlies the heterogeneity of means of dissemination by which research has societal impact. Within the 100 highest ranked publications, the different forms are:

• Academic commentary, which was most commonly published by the journals Science or Nature and usually involves a short summary of several pieces of research for lay people. For example, this piece provides an overview of research on the long-term effects of COVID-19.
• Academic reports, which are published by a university or other organization.
• Direct release of data, including data visualizations.
• Peer-reviewed articles in academic journals.
• Press releases which summarize research already published in a peer-reviewed article or preprint.
• Press releases which describe research not published elsewhere. For example, this UK government notification about the results of a trial on a treatment.

One publication was categorized as ‘other’, which was the release of a letter by the US Centres for Disease Control and Prevention on the use of Chloroquine Phosphate or Hydroxychloroquine Sulfate as a treatment.

Peer-reviewed articles in academic journals are the most common form of publication in the top 100. There were 31 direct links to articles or to press releases summarizing them. All the articles were open access, including those summarized in press releases, something which supports claims that open access helps encourage wider discussion of research among a lay audience.

The 18 preprints or press release summaries of preprints show that this relatively new form of publication does pose a real challenge to the traditional model of prioritizing publishing findings in a peer-reviewed journal article. Important research on COVID-19 is being published as preprints. For example, this research on under-reported COVID-19 cases was published in April and received a Reddit score of about five thousand seven hundred.

There were 17 academic commentaries in the top 100. These often took the form of short and accessible summaries of the most important literature, for example this letter published by a prestigious medical journal on the ‘Successful Elimination of Covid-19 Transmission in New Zealand’. These commentaries show the importance of researchers writing accessible literature reviews. Knowledge advances cumulatively and it is very rare that a single publication is truly ground- breaking. A key aspect of societal impact in the pandemic has been short summaries of available research.

There were four reports published by researchers in the top 100. This number perhaps doesn’t reflect the policy impact of publication via reports rather than peer reviewed articles. Especially in the early phases of the pandemic, reports published by bodies such as the World Health Organization, European and United States Centres for Disease Control and Prevention and national public health bodies played a key role in the developing public policy on the nature of the pandemic and how to deal with it.

Five entries in the top 100 concerned data, either the public release of raw facts or data visualizations. These represent a desire for an interested lay audience to be able to look at data directly rather than have it mediated via an article or other publication.

The most common entry in the top 100 with 25 entries were press releases which summarized ongoing research that was not published elsewhere. These press releases often publicized research carried out by private companies (for example this information about clinical trials for a vaccine). Press releases were also used to publicize research by government public health organizations. For example, ranked second with a score of nine thousand four hundred was this April summary of an antibody study by the Finnish Institute for Health and Welfare which found that that the ‘Number of people with coronavirus infections may be dozens of times higher than the number of confirmed cases’. In April, that was a very important finding. The importance of (initial) publication via press release underlines the fact that a large proportion of biomedical research is not conducted by university-based academics, a situation found in other areas of research (especially the social sciences).

Concerning peer review, it is important to note that research not published in academic journals is still commonly reviewed by peers before publication. Drafts of preprints, reports or findings presented in press releases will often already have been circulated for comments. However, at least in my experience these other forms of peer review will not have used the double-blind process employed by journal editors. Instead, for example, reviews may be conducted ‘in house’ with all parties knowing the identity of the other. I have found that such non-anonymous reviews can be the most useful. Reviewers whose identity is known have an incentive to provide timely and constructive criticism, though of course problems with bias remain.

To conclude, the mainly lay readers in the Reddit COVID-19 forum obtained their information from a wide variety of sources. Articles which have been through the double-blind peer review process are still important. But they amounted to less than a third of all the publications ranked in the top 100. In terms of societal impact, a press release might be more relevant than a journal article. One issue beyond the scope of this blog post is the quality of those sources. Certainly, the lack of peer review does appear to have allowed the dissemination of some poor-quality preprints (though I’m not referring to ones mentioned here), though there have been high profile failures in the peer-review process as well. In an age in which conspiracy theories and pseudoscience about COVID-19 have also achieved widespread societal impact it is important that lay readers have a simple means to distinguish quality research from bullshit. We can’t rely upon quality assurance from double-blind peer review when a majority of the most widely read scientific publications haven’t been through that process.

As a post-script, in the social sciences at least, peer-reviewed articles often have the reputation of being lengthy and turgid. I’ve been reading medical literature for many years, and this exercise underlined how much of the research is mercifully brief and where possible avoids jargon. Some of the peer-reviewed articles in the top 100 were shorter than this blog post, and their findings could be understood and discussed by people like me who lack a university degree in the biomedical sciences. Other fields should consider how expected forms of academic writing could affect societal impact, a consideration which is especially important given the success of the open access movement and calls for research to be accessible to a wide lay audience.

Nicholas Marsh is a Senior Researcher at PRIO. His research investigates the small arms trade using quantitative methods.