Let’s stop (usually) with the second round of review

I’m grumpy today about something that hasn’t even happened yet. Yes, that’s probably unreasonable; but I’m grumpy about something that happens too often, and I’m going to make myself feel better by venting just a little. I claim (at least partly because it’s true) that I have a real point to make.

Here’s what I’m grumpy about: second rounds of peer review. I have a manuscript with a journal that is (happily) pending a decision by the editor. So soon I’ll get reviews, and I know how this will go. First I’ll read the reviews and think they’re incompetent hack jobs. Then I’ll set them aside, re-read them a couple of days later and realize they’re actually highly perceptive and extremely useful.*  I’ll make revisions, the manuscript will be enormously improved, and I’ll resubmit. And here comes the part I’m pre-emptively grumpy about: the editor will send the manuscript back out for a second round of peer review. It’s happened to me at least half a dozen times in the last few years.

Can we stop with that part, please? It is, nearly all the time, a waste of precious resources.

I’ve been an associate editor, at a couple of different journals, for over 20 years, and I’ve handled a lot of papers. Early on, I committed the sin I’m now railing against: unless the revisions were truly trivial, when I got a revision I’d send it back out for further review. But I’ve come to understand that by consulting reviewers at this stage I was mostly trying to hide. I’m the editor; I should be able to judge whether the authors have responded appropriately to what the reviewers suggested in the first round. Not only can I make that judgement; I should, and I should own it.

It helped me, thinking about this, to realize that peer review has two different functions that are playing out at the same time. First, it has a manuscript-improvement function: peer review finds ways in which the science or the writing could be made better. Second, it has a gatekeeping function: peer review attempts to distinguish among manuscripts that should be published in the journal considering them; manuscripts that should be published, but in a different journal; and manuscripts that shouldn’t be published at all. These two functions aren’t shared evenly between reviewers and editors. While reviewers certainly comment on the gatekeeping side, that part is ultimately a decision made by the editor. (As a reviewer, understanding this, I’ve learned to devote the vast majority of my attention to manuscript improvement, especially because almost every manuscript is ultimately published somewhere.

As an editor, then, when I sent a revised manuscript out for a second round of review, I was refusing to make the decision that was my job to make – trying to hide, instead, behind whatever the reviewers would say. But if I used the same reviewers again, they’d already had their say; so I was implicitly asking them to find new issues to comment on – and authors were understandably upset. And if I used new reviewers, I moved the goalposts – and authors were understandably upset.

This might not matter all that much, in a perfect world. The second round of review makes papers better – just not as much better as the first round of review.**  The second round takes some time, but if I’m honest, not one of my papers has been so immediately world-changing that a couple of extra months in review would have significantly retarded the march of science.***

But there’s one big consideration that swings the balance strongly against second rounds of peer review, and that’s the limited capacity of the reviewer pool. Editors these days struggle to find willing reviewers. It’s not uncommon to hear of someone needing to ask 10 or 15 or even 30 people to secure two willing reviewers. We’re all busy, and the impacts on academia of the pandemic have left people burnt out and exhausted. With willing reviewers in short supply, it just doesn’t make sense to draw down that pool for a round of review that (usually) isn’t necessary.

So as an editor I’ve changed the way I operate. No matter how major the revisions are, I send out a manuscript for a second round of review only if I genuinely can’t judge whether the new version satisfies the suggestions from the first round of review. That may be true if, for instance, there are major new experiments, lots of new data, or an analysis so qualitatively different that I’m essentially dealing with a whole new manuscript. But these cases are rare. Ninety percent of the time (at least), I should be able to make the call without calling on reviewers again. After all, I’ve realized, if I can’t make that call I have no business being an editor anyway.

Perhaps in this particular case, my grumpiness will prove to have been misplaced. My fingers are crossed! Metaphorically, at least, because it’s hard to type with them really crossed.

© Stephen Heard  June 21, 2022


*^I’ve been through this cycle almost 100 times, and I’ve written about it, and you’d think knowing in advance how it’s going to go would help me avoid the “incompetent hack job” diagnosis. It doesn’t. Even knowing that I’m wrong, I still feel that way every single time.

**^Not because reviewers are tired of the manuscript and do a lesser job – although that might be true – but because the big improvements have already been made. Fisher’s geometric model of software updates applies analogously to manuscripts.)

***^This is even less of a concern given that many manuscripts are preprinted, including the one I’m pre-emptively complaining about.

8 thoughts on “Let’s stop (usually) with the second round of review

  1. Robin Pakeman

    I’ve done a lot of editing and my rule of thumb was to send it back to the original reviewers only where they asked for something highly technical to be addressed that was outside my own technical (i.e., statistical) competencies. So maybe 5 % of papers need that specialist second review to make sure an important change was done correctly. In all other cases I agree it is the editor who should make that decision straight away and not further burden reviewers. I would always make further recommendations to improve readability and presentation.

    Like

    Reply
  2. Peter Apps

    And isn’t it a huge pleasure when the second reviewers roundly criticise a change made at the insistence of the first, and tell you to change it back to what it was originally. Or when the second round disagree with both each other and the first round ?

    Like

    Reply
  3. Fiona Hay (IRRI)

    I entirely agree. Thank you for articulating this. I often decline to review a second time. Nor do I like it when I’m asked to review a paper that has already been reviewed by others. Maybe it’s also one of the reasons why it can be so hard to find willing first-round reviewers.

    Like

    Reply
  4. Eric Lamb

    Been there on the paper where we got a reject due to a new reviewer on the 2nd round commenting on “lack of potential impact”….

    That said, as an editor, I find a 2nd round of review very useful in a situation where the original manuscript had a lot of problems but also clearly a lot of potential, and where I am willing to work with the authors on it. In such cases the paper has often radically changed from the first version and I very much need a second set of eyes on it (I often will only ask one of the reviewers from the first round).

    Like

    Reply
  5. Chris Mebane

    Usually…. Respect to the reviewers also needs consideration. I have often put much time into a review pointing out what I considered correctable but major issues of interpretation, only to see the paper come out later without these changes. I presume to authors replied with a rebuttal that sounded plausible enough but was factually incorrect. As a reviewer, my response to the editor’s request for a second look is usually same day.
    But wearing my AE hat, I know this isn’t always the case. Original reviewers don’t respond or aren’t prompt. My compromise is sometimes to go outside the manuscript management system, email the reviewers manually with the responses and revisions and a note that I plan to act on it within 3 days unless I hear from them about the responses. Less than ideal, but somehow pushing it back through the ScholarOne system (for my journal) and its hard to override templates produces delays.

    Liked by 1 person

    Reply
  6. Scott

    Couldn’t agree more. While many of us may not be splitting atoms on a day to basis, having a paper mangled beyond recognition and waiting years for a final decision due to multiple rounds of peer review is such a waste of time for all of us. I recently had one go out to new reviewers 3 times after successive minor revision decisions and gave up once I passed 2 years from the original submission date.

    Liked by 1 person

    Reply

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.