I’m grumpy today about something that hasn’t even happened yet. Yes, that’s probably unreasonable; but I’m grumpy about something that happens too often, and I’m going to make myself feel better by venting just a little. I claim (at least partly because it’s true) that I have a real point to make.
Here’s what I’m grumpy about: second rounds of peer review. I have a manuscript with a journal that is (happily) pending a decision by the editor. So soon I’ll get reviews, and I know how this will go. First I’ll read the reviews and think they’re incompetent hack jobs. Then I’ll set them aside, re-read them a couple of days later and realize they’re actually highly perceptive and extremely useful.* I’ll make revisions, the manuscript will be enormously improved, and I’ll resubmit. And here comes the part I’m pre-emptively grumpy about: the editor will send the manuscript back out for a second round of peer review. It’s happened to me at least half a dozen times in the last few years.
Can we stop with that part, please? It is, nearly all the time, a waste of precious resources.
I’ve been an associate editor, at a couple of different journals, for over 20 years, and I’ve handled a lot of papers. Early on, I committed the sin I’m now railing against: unless the revisions were truly trivial, when I got a revision I’d send it back out for further review. But I’ve come to understand that by consulting reviewers at this stage I was mostly trying to hide. I’m the editor; I should be able to judge whether the authors have responded appropriately to what the reviewers suggested in the first round. Not only can I make that judgement; I should, and I should own it.
It helped me, thinking about this, to realize that peer review has two different functions that are playing out at the same time. First, it has a manuscript-improvement function: peer review finds ways in which the science or the writing could be made better. Second, it has a gatekeeping function: peer review attempts to distinguish among manuscripts that should be published in the journal considering them; manuscripts that should be published, but in a different journal; and manuscripts that shouldn’t be published at all. These two functions aren’t shared evenly between reviewers and editors. While reviewers certainly comment on the gatekeeping side, that part is ultimately a decision made by the editor. (As a reviewer, understanding this, I’ve learned to devote the vast majority of my attention to manuscript improvement, especially because almost every manuscript is ultimately published somewhere.
As an editor, then, when I sent a revised manuscript out for a second round of review, I was refusing to make the decision that was my job to make – trying to hide, instead, behind whatever the reviewers would say. But if I used the same reviewers again, they’d already had their say; so I was implicitly asking them to find new issues to comment on – and authors were understandably upset. And if I used new reviewers, I moved the goalposts – and authors were understandably upset.
This might not matter all that much, in a perfect world. The second round of review makes papers better – just not as much better as the first round of review.** The second round takes some time, but if I’m honest, not one of my papers has been so immediately world-changing that a couple of extra months in review would have significantly retarded the march of science.***
But there’s one big consideration that swings the balance strongly against second rounds of peer review, and that’s the limited capacity of the reviewer pool. Editors these days struggle to find willing reviewers. It’s not uncommon to hear of someone needing to ask 10 or 15 or even 30 people to secure two willing reviewers. We’re all busy, and the impacts on academia of the pandemic have left people burnt out and exhausted. With willing reviewers in short supply, it just doesn’t make sense to draw down that pool for a round of review that (usually) isn’t necessary.
So as an editor I’ve changed the way I operate. No matter how major the revisions are, I send out a manuscript for a second round of review only if I genuinely can’t judge whether the new version satisfies the suggestions from the first round of review. That may be true if, for instance, there are major new experiments, lots of new data, or an analysis so qualitatively different that I’m essentially dealing with a whole new manuscript. But these cases are rare. Ninety percent of the time (at least), I should be able to make the call without calling on reviewers again. After all, I’ve realized, if I can’t make that call I have no business being an editor anyway.
Perhaps in this particular case, my grumpiness will prove to have been misplaced. My fingers are crossed! Metaphorically, at least, because it’s hard to type with them really crossed.
© Stephen Heard June 21, 2022
*^I’ve been through this cycle almost 100 times, and I’ve written about it, and you’d think knowing in advance how it’s going to go would help me avoid the “incompetent hack job” diagnosis. It doesn’t. Even knowing that I’m wrong, I still feel that way every single time.
**^Not because reviewers are tired of the manuscript and do a lesser job – although that might be true – but because the big improvements have already been made. Fisher’s geometric model of software updates applies analogously to manuscripts.)
***^This is even less of a concern given that many manuscripts are preprinted, including the one I’m pre-emptively complaining about.