Reconciling the two functions of peer review

Image: Gandalf the Gatekeeper, CC 0 via goodfreephotos.com

Peer review is arguably central to what we do as scientists – to a considerable extent it’s what lets us recognize an authentic scientific enterprise.  Consider, for instance, the distinction between peer-reviewed publications and hack pieces in predatory journals; or think about how peer-reviewed grant proposals differ from pork-barrel politics.  Given this key role, it’s rather surprising to find a great deal of disagreement about what peer review is for, how it works best, or even whether it works at all.

Along these lines, I was very surprised a couple of weeks ago to see a flurry of tweets from some folks who wanted journals to give them a simple thumbs-up or thumbs-down on their manuscripts.  No comments, please, and no suggestions for improvement, thanks, just a writ of execution or an ennoblement. After all, one person pointed out, by the time they submit a manuscript, it’s already benefitted from plenty of comments – from coauthors and friendly reviewers.  More comments, they argued, would just tempt editors to reject what they thought was already a pretty good manuscript, thereby slowing down the publication process and, by extension, the march of science.*

My first reaction was that this was utterly bizarre, but after some thought, I can see the germ of a point in it.  What the “just the verdict, please” brigade is worried about (I think, although they didn’t say so) is a failure to reconcile the two functions of peer review.

Peer review** does two things for our literature.

  1. It plays a gatekeeping function. Peer reviewers make recommendations to a journal editor about whether the manuscript should be accepted as is, accepted after modification, or rejected. You can think of this gatekeeping either as a mechanism for keeping bad science out of the literature, or as a mechanism for assuring the quality and fit-to-scope papers published in a particular journal.  The fact that most manuscripts eventually get published somewhere suggests that the second flavour of gatekeeping is probably the more important one – although presumably there really is work so bad that nobody should publish it (you can insert here an appropriate joke about stuff that gets published away – just please stick with homeopathy and cold fusion, and leave my CV out of it).
  2. It plays a manuscript-improvement function. Comments from peer reviewers make published papers better than submitted manuscripts.  Drawing just on my own experience, they may suggest more powerful analyses, context from as-yet-uncited literature, clearer argument, fixes to analytical and logical errors, and any number of other enhancements.  Sure, not every reviewer comment is pure distilled wisdom; but even when a reviewer misreads or misunderstands a manuscript, that still gives an author an opportunity to adjust so that no other reader can misread or misunderstand in the same way.

Which of these functions is more important?  To me, it’s hands-down #2***.  Because most manuscripts will be published somewhere, making them better is where peer review makes a real difference to the literature.  But it’s clear that those in the just-the-verdict-please brigade disagree: not only do they think the manuscript-improvement function is unimportant, they’d prefer to do away with it entirely.  This opinion would be completely baffling except for one thing: the two functions, absent some care, can interfere with each other.  The risk is that reviewers may make comments in the belief that they see a way to make the manuscript better; but editors may interpret these as evidence that it isn’t currently good enough.  To the extent that this happens, it means the manuscript-improvement function is distorting the gatekeeping function.

I don’t know how often the two functions actually are in conflict, but I’m sure it’s not zero.  There are certainly glam journals that look actively for any reason to reject something (“glam” is something of a continuum, and I don’t think the phenomenon is restricted to Nature and Science). But there’s good news, too.  There are relatively simple steps that can be taken by authors, reviewers, and editors to reduce interference between manuscript improvement and gatekeeping:

  1. Authors: it’s a naïve view of the revision process to think that every comment by every reviewer must be accommodated, or else your manuscript will be rejected. That just isn’t true (and of course it can’t be when two reviewers contradict each other).  Some changes, to be sure, will be close to mandatory; others are mere suggestions; and a good chunk of the art of writing a strong Response to Reviews is your ability to sort the two out and make a clear case for how you’ve sorted them.  Writing the Response to Reviews, by the way, is a skill we ought to teach people (I do so in my scientific writing course, and I cover it in Chapter  24 of The Scientist’s Guide to Writing).
  2. Reviewers: remember that you can and should indicate which of your comments you feel are critical to address, and which are just suggestions. (That doesn’t mean the editor will agree with your categorization, of course, but at least you’ll have made your intentions clear).  There are lots of ways to do this.  I typically mark my most substantive comments with “***”, and place what I think of as my absolute-musts in a paragraph addressed to the editor before my more detailed comments.
  3. Editors: distinguishing between it-would-be-nice-to and this-manuscript-isn’t-publishable-unless might just be the most important part of the job. If the reviewers have made this clear, and you agree with them, that’s fine; but in the likely event that one or the other condition doesn’t hold, then the distinction must be made.  Simply tallying the votes on advice to accept or reject isn’t being an editor – a simple logic gate could do that!

In the end, I think the just-the-verdict  brigade is operating from some naïveté about how peer review works and how editors do their job, coupled with some reasonable fear that peer reviewers and editors may not always do their jobs perfectly.  But with easy fixes to be had, it would be foolish of us to give up on the opportunity to make our work better****.

© Stephen Heard  March 25, 2019


*^I’m not going to identify the folks I’m paraphrasing, but I promise I’m not making this up.

**^I’m restricting discussion from here on to peer review of manuscripts for journals

***^I suppose one could argue that from the point of view of a journal, faced with an inexhaustible supply of manuscripts, it might be #1, as one could in theory simply reject everything short of perfection.  One could argue it, although I think it would be a weak argument.  But from the perspective of authors and of readers, I can’t see any pro-gatekeeping argument at all.

****^It occurred to me mid-writing that the belief that peer review should only gatekeep might explain what is otherwise an extremely puzzling phenomenon: the habit some authors have of submitting a rejected manuscript to a new journal without making any changes.  This seems foolish to me: surprisingly often, rejected and resubmitted papers find their way to the same reviewers again – and they get the rough ride you’d expect.  But if you think your paper is already as good as you can make it, and only want to find a journal for which it passes gatekeeping, I suppose it’s not entirely foolish.  No, never mind; it is entirely foolish – but perhaps it’s foolishness of an explainable origin.

 

Advertisements

3 thoughts on “Reconciling the two functions of peer review

  1. thetweedybiologist

    I think the problem is function #2 is an emergent property of peer-review. If manuscript improvement were the proximate goal, then rejections would be decidedly rare and would happen only after several rounds of revisions in which the authors do not substantially revise the manuscript (perhaps journals would look more like arXiv in such a scenario). As authors (and possibly reviewers and editors), function #1 is what is most visible to us which is what I think drive the “thumbs-up; thumbs-down” review process.

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      Yes, interesting point – historically, the direct and explicit rationale for adding peer review was probably gatekeeping rather than manuscript improvement. And so people can be pardoned for not noticing that manuscript improvement is actually more important. Of course, there are lots of other examples where an incidental byproduct of Thing X is now the primary function of Thing X.

      I’d push back a little, though, on “what is most visible to us”. Sure, the reject-or-accept may be in the first line of the letter you get from the editor. But it can’t take too much digging to discover the suggestions for improvement, can it? I think that’s what I’m getting at in my suggestion for authors – gatekeeping is only the most obvious and important part if you decide that it is. I’m probably distorting what you meant to say, though.

      Thanks for commenting!

      Liked by 1 person

      Reply

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.