Image: Author expectations for “optimal” peer review: Figure 1 from Nguyen et al. (2015) PLoS ONE 10(8):e0132557.
Two things I saw last week motivated todays’ post. The first was Amy Parachnowitsch’s interesting blog post, wondering if peer review might sometimes be faster than she’d like: too fast for her to get head-clearing perspective by putting a manuscript away for a while. The second was a paper by Nguyen et al. reporting author opinions of how long peer review should take. Some of those opinions are absolutely astonishing.
I’ll get my astonishment in a moment, but first: how long should peer review take? I’m going to assume here that we all understand that peer review has two functions: a gatekeeping function (passing judgement on whether a paper should be published at all) and a manuscript-improvement function (providing comments that make the published paper better than the submitted manuscript). I’m going to assert that despite its occasional warts, the peer review process overall performs both these functions very well, and that we want it to continue performing both functions. So what kind of time should it take to get a manuscript through peer review?
I’m going to take a typical journal’s workflow, and guesstimate the minimum reasonable times we ought to allow for each step in the process*. This is not to say a given step couldn’t happen faster on occasion, but my guesstimates are such that I think those occasions will be rare. Thus, it would be patently unreasonable to expect anything faster while still allowing peer review to perform its functions. Here goes. Upload your manuscript, punch “submit”, and then allow:
-
1 business day** for the journal office to process your submission (format checks, etc.)
- 3 for the Editor-in-Chief (EiC) to do a quick read and assign to an associate editor (AE)
- 3 for the AE to read the manuscript, decide if it should go to review, and invite reviewers
- 7 for the journal office to contact reviewers and for enough to agree to review
- 10 for the reviewers to complete their reviews
- 1 for the journal office to process the reviews and notify the AE
- 5 for the AE to consider the reviews, read the MS more thoroughly, and write their own review and decision recommendation
- 4 for the EiC to read the MS, consider the AE’s recommendation, and make a final decision
- 1 for the journal office to finalize the paperwork and issue the decision letter
This comes to 35 business days, or a bit more than 7 weeks (depending on the luck of the statutory-holiday draw). If some of the individual times seem long, remember that everyone except possibly the journal office staff is unpaid and doing their journal work on top of all their other responsibilities. Sure, I can (sometimes) review a manuscript in half a day; but when I receive one to review, it goes in the queue with everything else on my plate. There is, after all, no reason the review I was asked for this morning is more important than the six thesis chapters my students sent me last week.
So 7 weeks is, I think, the absolute minimum that it’s reasonable to expect – not the average, but the minimum. And while online technology has drastically cut peer-review times over the last decade or two (when I was a grad student, 16 weeks was about the best you could hope for), I don’t think any amount of technology will cut them any further. The constraints now aren’t systems-related; they’re process-related. The process, I think, has to take about that long if we want it to work. This means, by the way, that journals that compete on claims of peer-review turnaround faster than 7 weeks are either promising something they can’t deliver, or abandoning the quality that comes with serious peer review. (Enhancements like transferability of peer reviews may cut time for subsequent consideration of the same manuscript, but I don’t see potential for major cuts in the first go-around. And we hear occasional calls for a complete reimagining of the whole journal-publication system, but I have yet to see a convincing case for this.)
So, peer review is unlikely to ever be faster than we’d like (you’re welcome, Amy). Is it slower than we’d like? Here’s where the Nguyen et al. paper comes in; prepare to be as astonished as I was. Nguyen et al. surveyed authors in conservation biology***, asking them questions about typical peer-review times (defined as everything from submission to decision, as in my breakdown above), and about what they considered short, optimal, and long review times. The results blew me away.
What do authors want? Their average “optimal” review time (Nguyen et al.’s Figure 1, reproduced above) was six weeks – a week less than the 7 weeks I think is the minimum reasonable time. Five respondents thought one week was optimal (and, presumably, each of them also wanted a pony).
What do authors consider to be “slow” peer review? On average, 14 weeks – but a significant number of authors have stricter standards. Fifty (a full 10% of respondents) thought a “slow” review meant 7 weeks or less! One respondent said one week was a “slow” review – demonstrating beyond reasonable doubt either that some people like to punk survey studies, or that some people are remarkably naïve****. The non-trivial number of votes for 2 and 3 weeks suggests, unfortunately, the latter.
I just can’t wrap my head around these expectations. Look: peer review has a job to do, and doing that job takes time. It would take time even if the whole process was handled by full-time, paid professionals; but it’s largely handled by volunteers who are also busy with other things. Peer review that takes a couple of months is not evidence of a broken system, and isn’t a reason we should be looking for alternatives. I, of course, believe that each and every one of my papers is an important advance for science (grin), but I think the world can generally wait a couple of months while peer review makes them better.
There is, it’s true, pressure on the peer-review system, with submission volumes rising continuously and faster than the pool of available reviewers (and AEs). Unreasonable expectations about the speed of the process make this pressure worse. An obvious way to alleviate that pressure is to take everyone who thinks a 7-week peer review process is “slow”, and invite them to review an extra manuscript each month – or to sign on as an AE. We can use the help.
UPDATE: See a closely related post from Terry McGlynn, who independently comes to a quite similar analysis.
UPDATE#2: For more grumpiness on the subject, this time aimed at journal rather than authors, see Things that are more important to me than reviewing your manuscript.
© Stephen Heard (sheard@unb.ca) August 31, 2015
Related posts:
*^I think my guesstimates are pretty good for the steps I have experience with (reviewer and associate editor steps). They’re probably underestimates for the steps I know less about (journal office and editor-in-chief); at least, I always assume I’m underestimating how hard somebody else’s job is.
**^Yes, I frequently work weekends and holidays, including handling and reviewing manuscript; but I’m not proud of it. And you certainly don’t get to assume that I’m going to work a holiday to handle your manuscript!
***^Whether other authors would have given similar answers is unknown; but based strictly on anecdote, I’d expect lots of fields to be even less patient.
****^“Naïve” was not the first word that sprung to mind, but I’ll give a little benefit of the doubt.
Scientist Sees Squirrel 
Striking paradox number one is that when sent an e-mail survey about a time consuming voluntary activity, only about 10% volunteered to respond.
LikeLiked by 2 people
Peter – nicely spotted! My irony-meter must have been overwhelmed with other stuff; I wish I’d noticed that myself…
LikeLike
Well that your student chapters are more important is not a justification to explain week’s end weeks of waiting time. Because then I can feel that my paicological health is more important or give water to my plants. All this delay can’t simply be included as reasonable waiting time. Sorry but if you have to calculate the optimal time has to be real optimal time.
LikeLike
If you think the reason I won’t review your paper in a week is that I’m “wasting time”, I think you have a rather poor understanding of what’s on my plate, and what you’re entitled to ask of me. And, by extension, any of our colleagues. Sorry to be blunt.
LikeLike
Academics get paid to train students, they don’t get paid for reviewing, and so it makes perfect sense for them to put their students first. I am not an academic and so, if I were to be given a paper of yours with grammar and spelling like that, I could offer you a review time as short as you could desire – no longer than it would take me to find the reject button on the reviewers’ dashboard.
LikeLike
Agreed, some of those views are astonishing. But I wonder if these unrealistic expectations are being driven by both advances in IT (as you suggest) and unrealistic review deadlines by journals.
You’ve allocated 10 days to that part of the process and (anecdotally) that seems to me to be an increasing expectation of journals: “Please complete your review within 10 days”. That can be a hard turn-around time to achieve given the other time pressures of the job. But if journals have that expectation, perhaps that’s (subconsciously) becoming an author expectation too, that their review will be completed super-quick?
LikeLike
Yes, I agree. I think 10 business days (2 weeks) is a bare minimum, given everything else we have to do. I’m much happier as a reviewer (and much more likely to say yes) given 3 weeks.
LikeLike
I serve as editor-in-chief of a journal, and we’ve recently changed our request to within 3 weeks, whereas it was 2 weeks for years. The thought here was that reviewers were more likely to say “no” the shorter the time frame for getting the review done. It is still tough to get reviewers to agree, but I think this is working. Haven’t actually run the numbers yet, though
LikeLiked by 2 people
I’d never really thought about all the processes needed as part of peer review, so I’ll bear that in mind when waiting to hear back about my next MS.
Do you have any feeling as to whether the quality of reviews relates to time taken to review a paper? My limited experience is that the papers I have heard back about quickest have had the most thorough reviews. But maybe this is because journals with higher impact factors are more demanding of their reviewers?
LikeLike
Speed vs. quality is a good question. In the extreme, of course, you can’t expect both quality and a 5-day turnaround. But with more reasonable times, maybe – but as an empirical question, I don’t have data!
LikeLiked by 1 person
I suspect that there is very little correlation here. I’ve never really done an analysis–I do suspect that those reviews that I have to harass and harangue to get delivered might be a little skimpier. But mostly, good reviewers write good reviews and sometimes they have a lot on their plate. As an editor, my ideal reviewer is early to mid-career–someone with knowledge but without as full a plate. I wrote a blog post about how to recommend reviewers: https://schimelwritingscience.wordpress.com/2015/10/02/how-to-recommend-reviewers-when-you-submit-a-paper/
But e
LikeLiked by 1 person
Oops. I cut myself off. But equally, I was going to say, authors need to remember that journals are “us”–their peers and colleagues. Maybe the person they might want to postdoc with. Remember the Golden Rule: treat others as you would wish them to treat you. The should include having a little patience.
LikeLike
Josh, that’s a really great post on recommending reviewers.
Everyone else: read Josh’s post, it’s really useful!
LikeLike
Pingback: Link Round-Up: Academic Journals and the Publishing Industry
Pingback: Is everything “broken”? | Scientist Sees Squirrel
Pingback: New paper published – Biological and climatic controls on leaf litter decomposition across European forests and grasslands revealed by reciprocal litter transplantation experiments | Ülo Niinemets’ Lab
Pingback: Post-publication peer review and the problem of privilege | Scientist Sees Squirrel
Hi Steve,
Thanks so much for writing this – it’s so important that the value of patient, careful peer review be pointed out. Your time estimates are very good, and certainly give the approximate minimum time in review. In my experience of running Mol Ecol, we’ve seen the biggest delays at the “journal office to contact reviewers and for enough to agree to review” stage. Some papers require many (>20) review requests to find at least two reviewers, and if people ignore the request (rather than just declining) the journal ends up waiting a week each time before moving to the next person.
The other big delay occurs when only two reviewers have agreed and one goes AWOL, as the journal must decide whether to give up on them and find someone else (which entails at least an additional 2-3 week delay), or continuing to badger the overdue reviewer. Make the wrong choice and you can add months to the decision time.
LikeLike
Hi Tim – yeah, your experience matches mine; those are two major reasons for delay. These are (of course) reasons I describe my timetable as the reasonable minimum!
Is your sense, like mine, that the time to get enough reviewers is going up, as reviewers are declining more?
LikeLike
We’ve just finished collecting a detailed dataset on whether the behaviour of reviewers has changed over my six year tenure at Mol Ecol, and we’re now trying to make sense of it. Hopefully we’ll be able to answer this question in the new year.
LikeLike
Pingback: Impatience with the peer review process | Small Pond Science
“3 for the Editor-in-Chief (EiC) to do a quick read and assign to an associate editor (AE)”
the words “quick read” and “3 days” certainly don’t match up. You’ve listed the maximum, for instance if the EiC was busy (you may have done that also for the other steps. 1 day to write a decision letter? come on..).
LikeLike
I’m not sure I understand you. Are you suggestion you don’t need 3 days to read and assign? Or don’t need a full day to write a decision letter? Sure, if your only job was being EiC, and there wasn’t a queue of MSs waiting, you might be right. But in the real world, I have a lot more going on, and those timelines are pretty conservative. (If I’ve misunderstood your point, let me know).
LikeLike
You seem to think that Editors are sitting and hovering over their in-boxes, waiting to jump on your paper the minute it appears. I try to check my in-box every few days to deal with the quick stuff, but otherwise, I often take Saturday morning to try to catch up on editorial work (on that issue I disagree with Stephen Heard–what’s a “business day”?). So yeah, it might only take me 10-20 minutes to decide whether to send a paper out for review and then to go through the database to select reviewers to invite, but hey, if your paper arrived in my in-box on Monday, it could sit there for a week before I give it that 10 minutes! And no, I don’t consider that slow. The people invited to be editors are people who are knowledgeable and experienced–they have lives and careers; being a journal editor is one small part of that. So have realistic expectations of your friends and colleagues who do these jobs as a service to the community.
LikeLiked by 1 person
Pingback: The one kind of review that really gets my goat | Scientist Sees Squirrel
Pingback: Is this blog a “science blog”? If not, what is it? | Scientist Sees Squirrel
Pingback: Things that are more important to me than reviewing your manuscript | Scientist Sees Squirrel
I’m a Chief Editor of Soil Biology & Biochemistry and we don’t work with a single Chief Editor who has to oversee Associate editors, so we might save a few days in some of those steps, but I would note that the long steps in Heard’s analysis: First 7 days to get reviewers to agree? I’ve sometimes taken months to get enough reviewers to respond to requests. All too often, people don’t respond to requests so the request sits with them for a week and then we pass along to the next down the line. I’ve handled papers where 10 people or more either never responded to the request or they declined. At least when someone declines quickly it allows us to move on quickly. But I can’t invite 6 people right away hoping that 2 or 3 will respond–what would I do if they all said yes. So if you want speedy review–respond to requests and respond quickly. Second, we officially give reviewers not 10 days, but 21 to provide a review. And many (including myself sometimes) don’t get a review in that quickly. It is common to have to remind reviewers to get their reviews in on time. So the two steps that take the most time in the process lies with us, us as researchers and reviewers, rather than with us as editors or journals. If you want quick review, you must commit to responding and reviewing quickly. Don’t blame the journal. We have met the enemy and he is us.
So yes, I absolutely agree with Stephen about a month being a reasonable absolute minimum. I’ve occasionally managed to get a paper reviewed and back to the authors in less than 2 weeks, but that’s one out of many hundreds I’ve handled. Given the vagaries of getting reviewers to agree to review and then getting reviews back in hand 2 months is a more common practical “reasonable” target.
LikeLiked by 1 person
Pingback: A month of spamvitations | Scientist Sees Squirrel
Pingback: Drop everything and check your page proofs, or else | Small Pond Science
Pingback: Why journals like “reject, but resubmit” | Scientist Sees Squirrel
Pingback: I refuse all review requests with deadlines < 3 weeks. Here’s why, and how. | Scientist Sees Squirrel
Pingback: How do you choose a journal when it’s time to submit a paper? | Scientist Sees Squirrel
Pingback: How long should you wait before nudging the journal about your manuscript in review? | Scientist Sees Squirrel
Pingback: Yes, most reviews are submitted at the deadline. No, that doesn’t justify shorter deadlines | Scientist Sees Squirrel
Pingback: Eight is (more than) enough: How peer review gets out of hand | Scientist Sees Squirrel
Pingback: Should we boycott journals that use bullying tactics to speed up their review process? | Don't Forget the Roundabouts
Pingback: How much damage do retracted papers do to science before they’re retracted, and to whom? | Dynamic Ecology
Pingback: Unclogging the peer review pipeline - Fierer Lab
Pingback: Those journals may be “fake”, but I don’t think they’re “predatory” | Scientist Sees Squirrel