Eight is (more than) enough: How peer review gets out of hand

A colleague recently mentioned being astonished to receive eight different peer reviews, on a single manuscript in a single round of reviews at a single journal.*  Wasn’t this too many, he asked?  And how could it happen?

Well, I’m here to serve.  Yes, eight is too many.  As for “how could it happen”: that’s a bit more complicated, but I’ll give you a plausible guess.

First: eight is too many, but not for the reason you might think.  It’s tempting to believe it’s because with eight reviewers, it’s almost inevitable that every suggestion made by Reviewer A will be contradicted by one of Reviewers B, C, D, E, F, G, or H.  That will happen, true; but it really isn’t a problem.  I used to find it frustrating when reviewers disagreed (how could I possible make a change that will satisfy them all?), but I’ve come to understand that it actually puts the power right in my lap. If one reviewer wants X, and another wants not-X, then I can do X or not-X as I prefer (as long as I justify my choice in the Response to Reviews.  With eight reviewers I can probably justify almost anything by pointing to one review or another.  (I should, of course, use that power for good and not for evil: to help me settle on whatever revision is best, not to justify whatever revision is easiest.) 

So if the problem it isn’t inter-reviewer conflict, what is it?  It’s simple: inefficiency.  It’s hard for an editor to find willing reviewers, and using eight for one manuscript wastes a limited commodity.  I say “wastes”, because inevitably as reviews pile up the marginal value of each additional one decreases.  That eighth review took just as much reviewer effort as the first one, but because it’s likely to repeat points from the first seven, it probably won’t improve the manuscript much.

Now, how could this happen?  I don’t know for sure, of course, but I can make a pretty good guess.  Ironically, what drove this particular bout of over-reviewing was almost certainly the very fact that it’s hard to get reviewers – coupled with the fact that unrealistic expectations on the part of authors is driving journals to compete based on speed.  Let me explain.

When I’m serving as Handling Editor** for a manuscript, my goal is normally to procure two reviews, or maybe three.  To achieve this, the manuscript-handling system will have me make a longer list of possible reviewers – at least six, and often more.  That’s because many reviewers will decline their invitation to review.  There are two obvious systems one could use to get two reviews from six possible reviewers.  The better one is to invite the first two reviewers on the list, and then if one declines, to invite the next one on the list.  We can call this sequential invitation.  The other system is to invite the whole list at once, in the hopes that any two will agree.  This is simultaneous invitation.  So now you can see how a single manuscript could get eight reviews: the journal used simultaneous invitation, and the whole list, much to everyone’s surprise, said yes.***  Eight reviewers are thus the equivalent of an overbooked airline flight.

But isn’t simultaneous invitation, with its risk of over-reviewing, an exploitative waste of reviewers’ time?  Yes, it is.  But (some) journals do it because it’s faster.  That matters because many journals compete for submission by advertising their quick time-to-first decision.  And why do journals compete based on speed?  Because of authors.  Many authors seem to have unreasonable expectations for how fast peer review can be done – wanting to submit a manuscript and have an answer in five weeks, or three weeks, or astonishingly, in one week.  And so journals use simultaneous invitation to shave days from their decision time (and then further abuse reviewers by setting unreasonably tight deadlines for the receipt of reviews).  Eight reviews isn’t an outcome that authors want, and isn’t an outcome that journals want.  But it’s an unintended consequence of journals trying to give authors what they’re asking for.

It would be nice if we could all do small things to prevent the eight-reviewers inefficiency.  Fortunately, we can.  If you’re an editor and have a choice, use sequential invitation.  If you’re an editor and your journal currently requires simultaneous invitation, push back.  And if you’re an invitee, accept or decline an invitation to review right away.  Don’t be pushed into reviewing right away – no journal gets to jump the queue of work that’s no doubt waiting for you – but if we were all quick to agree or decline, journals would find less reward in simultaneous invitation.  Quick to respond, slow(er) to review – that, I think is the sweet spot.

© Stephen Heard  March 10, 2020

Image: OK, not really all the reviewers for a manuscript.  Really, meeting of the  Federal Open Market Committee, Washington, DC.  Public domain.


*^In some online discussion, it turned out that this might set a record, but it doesn’t smash the old one.  Another person reported getting seven, and yet

**^Also known, depending on the journal, as Associate Editor, Subject Editor, and I’m sure I’ve forgotten a synonym or two.

***^If three-quarters of all reviewers say no, then a simultaneous invitation of six will yield six reviews about 0.02% of the time (0.25 to the sixth power).  Or if half say no, inviting eight will yield eight reviews about 0.4% of the time.  Reviewer agreement rates vary across journals, and you can play with the numbers; but the everyone-says-yes case will be rare but not unheard-of.

13 thoughts on “Eight is (more than) enough: How peer review gets out of hand

  1. Michael Wiener

    I’ve seen another reason for getting many reviews: changing the accept/reject decision. Usually, when the person handling the paper wants a particular outcome, he/she can select reviewers likely to give that outcome. When that fails, get more reviewers to justify the desired outcome.

    Like

    Reply
  2. Daniel Weissman

    I think simultaneous invitation is fine. If too many people say yes, the editor can always email the later ones and say, “Actually, it looks like we already have enough. Can you hold off for now, and we’ll be back in touch if one of the earlier people flakes out?” This has actually happened to me, and unlike with the overbooked airplane, I was pretty happy to get the email.

    Like

    Reply
    1. Juan Antonio CARRETERO

      This can even be automated. You can send the simultaneous invitation to as many reviewers you may reasonably want but the system could automatically send a nice “cancel invitation” to those who have not answered the invitation the moment the minimum number of “agrees” is met. I, for one, do not get upset when this happens, much the opposite as it means less work on my plate.

      Like

      Reply
  3. Paul Abram

    I found your probability calculation in the footnotes instructive! do also ask myself now WHY everyone said yes, especially because the review went out over the holidays. Hopefully because it was of interest, but maybe just a roll of the 8-sided die…

    Like

    Reply
  4. Pavel Dodonov

    I’m curious, how common is it for reviewers to take long to accept/decline an invitation? I mean, for me it’s basically “see the email – check my agenda to see if I have time – respond”. If I’m at the work place, it’s almost instantaneous. Otherwise I respond when I’m back at the office, which doesn’t usually take more than three days. I don’t see many logical reasons to take too long a time to accept/decline an invitation…

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      You’re one of the good ones! Unfortunately, it seems to be fairly for someone to hem and haw about an invitation for a week – sometimes two – before declining. Or to not even reply at all – and if you just ignore the invitation, the editor has no idea what’s going on. For MOST journals, declining is a couple of mouse clicks (a few more if you’re willing to suggest alternative reviewers).

      Like

      Reply
  5. cinnabarreflections

    One complication is that invitations sent out by an editor through editorial software sometimes end up in the spam folder. That contributes to the non-responses, but how much I don’t know. Your final sentence is right on: “Quick to respond, slow(er) to review – that, I think is the sweet spot.”

    Liked by 1 person

    Reply
    1. Chris Mebane

      I think the spam filter is a widespread problem, but unrecognized by editors because we’re used to a lot of non-replies as it is. At a conference, I approached an author whose paper I had handled to let them know I was disappointed they were not willing to pay it forward and hadn’t even bothered to respond to two review requests for related manuscripts. ‘Review requests? What review requests? I don’t remember review requests from you. I would be happy to review.’ I then started checking my own spam folder more and have found several requests relegated there.
      Solution? Sometimes I’ve followed up with a personal note (not sent through the editorial software!) asking if they saw the request and that I’d appreciate them letting me know if they can review. I have people say they didn’t see it, or I might get an out of office message that helps. But I have to remember to do so and I have to remember to copy myself on the original invite in the first place. So it’s not a very good solution.
      Any other ideas for not getting flagged as Review Spam?

      Like

      Reply
  6. Peter Erwin

    That’s… genuinely kind of amazing. (It reminds me of the class a year ahead of me in grad school, which was about three or four times too large because *everyone* who was accepted chose to come, instead of the 1/3 or so.)

    Admittedly, in my field (astronomy) we get by with just *one* reviewer per paper, and so we think it reasonable for the submission-to-receiving-review process (from the authors’ perspective) to take about a month or a little more. (I think our journals may have less overhead, because I think most acceptance decisions are made by the individual Scientific Editors — probably equivalent to the Associate Editors in your case — so there’s no need to write internal reports and have the Editor-in-Chief approve everything. I might be wrong about that detail, and there is the one journal that interposes “assistant editors” in between the Scientific Editor and the authors.)

    Liked by 1 person

    Reply
  7. Manu Saunders

    Wow, that’s terrifying as an author. Maximum number of reviews I’ve ever received is 4, and that was stressful enough to respond to. If I received 8 reviews, I would email the editor to request some negotiation on what I really need respond to in revisions.

    As an editor, I’ve never invited 8 simultaneously and I don’t understand how/why that could happen…It didn’t even occur to me that some journals would allow this! My required minimum reviews at all journals i’ve worked with are 2, and I always wait until I get a decline/uninvite before I invite another reviewer (which obviously drags the review timeline out). A few times I end up with 3 reviewers (which I actually think should be the minimum), usually when I’ve had so many declined invitations that I get desparate to get the paper ‘under review’ and I invite a couple of people at a time.

    Like

    Reply
  8. Pingback: Science in the ol’ days: A millennial’s perspective – Brushing Up Science

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.