A colleague recently mentioned being astonished to receive eight different peer reviews, on a single manuscript in a single round of reviews at a single journal.* Wasn’t this too many, he asked? And how could it happen?
Well, I’m here to serve. Yes, eight is too many. As for “how could it happen”: that’s a bit more complicated, but I’ll give you a plausible guess.
First: eight is too many, but not for the reason you might think. It’s tempting to believe it’s because with eight reviewers, it’s almost inevitable that every suggestion made by Reviewer A will be contradicted by one of Reviewers B, C, D, E, F, G, or H. That will happen, true; but it really isn’t a problem. I used to find it frustrating when reviewers disagreed (how could I possible make a change that will satisfy them all?), but I’ve come to understand that it actually puts the power right in my lap. If one reviewer wants X, and another wants not-X, then I can do X or not-X as I prefer (as long as I justify my choice in the Response to Reviews. With eight reviewers I can probably justify almost anything by pointing to one review or another. (I should, of course, use that power for good and not for evil: to help me settle on whatever revision is best, not to justify whatever revision is easiest.)
So if the problem it isn’t inter-reviewer conflict, what is it? It’s simple: inefficiency. It’s hard for an editor to find willing reviewers, and using eight for one manuscript wastes a limited commodity. I say “wastes”, because inevitably as reviews pile up the marginal value of each additional one decreases. That eighth review took just as much reviewer effort as the first one, but because it’s likely to repeat points from the first seven, it probably won’t improve the manuscript much.
Now, how could this happen? I don’t know for sure, of course, but I can make a pretty good guess. Ironically, what drove this particular bout of over-reviewing was almost certainly the very fact that it’s hard to get reviewers – coupled with the fact that unrealistic expectations on the part of authors is driving journals to compete based on speed. Let me explain.
When I’m serving as Handling Editor** for a manuscript, my goal is normally to procure two reviews, or maybe three. To achieve this, the manuscript-handling system will have me make a longer list of possible reviewers – at least six, and often more. That’s because many reviewers will decline their invitation to review. There are two obvious systems one could use to get two reviews from six possible reviewers. The better one is to invite the first two reviewers on the list, and then if one declines, to invite the next one on the list. We can call this sequential invitation. The other system is to invite the whole list at once, in the hopes that any two will agree. This is simultaneous invitation. So now you can see how a single manuscript could get eight reviews: the journal used simultaneous invitation, and the whole list, much to everyone’s surprise, said yes.*** Eight reviewers are thus the equivalent of an overbooked airline flight.
But isn’t simultaneous invitation, with its risk of over-reviewing, an exploitative waste of reviewers’ time? Yes, it is. But (some) journals do it because it’s faster. That matters because many journals compete for submission by advertising their quick time-to-first decision. And why do journals compete based on speed? Because of authors. Many authors seem to have unreasonable expectations for how fast peer review can be done – wanting to submit a manuscript and have an answer in five weeks, or three weeks, or astonishingly, in one week. And so journals use simultaneous invitation to shave days from their decision time (and then further abuse reviewers by setting unreasonably tight deadlines for the receipt of reviews). Eight reviews isn’t an outcome that authors want, and isn’t an outcome that journals want. But it’s an unintended consequence of journals trying to give authors what they’re asking for.
It would be nice if we could all do small things to prevent the eight-reviewers inefficiency. Fortunately, we can. If you’re an editor and have a choice, use sequential invitation. If you’re an editor and your journal currently requires simultaneous invitation, push back. And if you’re an invitee, accept or decline an invitation to review right away. Don’t be pushed into reviewing right away – no journal gets to jump the queue of work that’s no doubt waiting for you – but if we were all quick to agree or decline, journals would find less reward in simultaneous invitation. Quick to respond, slow(er) to review – that, I think is the sweet spot.
© Stephen Heard March 10, 2020
Image: OK, not really all the reviewers for a manuscript. Really, meeting of the Federal Open Market Committee, Washington, DC. Public domain.
*^In some online discussion, it turned out that this might set a record, but it doesn’t smash the old one. Another person reported getting seven, and yet
**^Also known, depending on the journal, as Associate Editor, Subject Editor, and I’m sure I’ve forgotten a synonym or two.
***^If three-quarters of all reviewers say no, then a simultaneous invitation of six will yield six reviews about 0.02% of the time (0.25 to the sixth power). Or if half say no, inviting eight will yield eight reviews about 0.4% of the time. Reviewer agreement rates vary across journals, and you can play with the numbers; but the everyone-says-yes case will be rare but not unheard-of.