How (as an editor) I choose lists of reviewers

Image: The reviewer-selection screen at one journal I edit for.

Warning: more detail than you may care for. 

Every manuscript submitted to a (peer-reviewed) journal needs reviewers, and it’s the editor’s job to choose appropriate ones.  How does that happen?  Have you wondered? Well, I can’t tell you how it happens in general; but I can tell you how I do it.

I’m an associate editor for two journals*The American Naturalist and FACETS.  Each time I’m assigned a manuscript to handle, I need to decide whether to send it out for review.  If the answer is “yes”, then I need to identify a list of possible reviewers, who will be invited, sequentially, to review until two (normally) accept.  For the Naturalist, I’m asked to make a list of 6 possible reviewers.  Usually we succeed in getting two of those 6 to review, but sometimes I need to identify more**. How do I choose names to include, on my first list or an expanded one?

I don’t have a complete rigid system, but here’s a typical workflow.

  • To end up at a list of 6 reviewers, I’ll aim for a scribbled-on-paper list of 10-12. That’s partly because inevitably a couple of my brilliant ideas turns out to be unavailable for various reasons.  More importantly, it gives me some easy room to pick 6 of the 12 in a way that roughly optimizes a combination of fit and reviewer diversity – rather than being constrained to the first 6 I think of.
  • My first move is to consult the authors’ list of suggested reviewers. Authors know their subfields well, and in my experience usually) make good suggestions.  I’ll almost always add 1-3 author suggestions to my list, although with the caveat that when we issue invitations from the list of 6, I want to use only one of the authors’ suggestions, and one of mine.  You might think I’m trying to prevent authors from gaming the system by suggesting only their friends.  My “1 of theirs, 1 of mine” rule does prevent that, but that’s not its primary purpose.  Instead, I’m aiming to represent perspectives the authors might not have thought of.  Those could be diversity perspectives – when authors name a bunch of senior people, those lists can be quite unrepresentative – or they could be disciplinary or organism-centric perspectives (the latter may be the most common thing I’m trying to broaden).  I think peer review is more useful if the reviewers are thinking from different places.
  • I’ll also, of course, consult the authors’ list of “opposed” reviewers (at submission time, the authors can list people that they don’t want to review the MS). Almost all the time, I respect that list; there are enough wise reviewers in the world that it just isn’t a problem to do that.  I don’t guarantee to respect the list, though (most journals don’t).  And authors proposing more than a couple of opposed reviewers will earn a little bit of side-eye and a higher probability that I’ll overrule them – unless, of course, they provide a convincing explanation***.

  • My next step is usually to add a few names that come to my own mind as suitable reviewers. When I began editing, I thought this would be my most important step; but I now understand that it’s my least  It’s rare that I let more than one of these “recollected” names crack my final 6.  However, doing this does get me thinking about the paper’s topic – and also other topics to which it might connect.  That helps me find other reviewers in my next steps.  This minimal use of recollected names is important.  It helps me minimize implicit biases, because like probably everyone else, I do a better job of representing diversity if I use recognition, not recall.  By “representing diversity” I mean diversity of all kinds: gender, career stage, academic/nonacademic affiliation, you name it.
  • Next, I’ll add some names garnered from papers cited in the manuscript. I’m reading the manuscript anyway, at this stage, so it’s just a matter of noting citations that seem particularly key, or – and I almost always do this – skimming the Lit Cited section after I’m done for titles that are a propos.  Quite often this yields a couple of names that provoke a “gosh, why didn’t I think of them?”.  Almost always, it yields a few names I wouldn’t have thought of.
  • I don’t usually have enough names at this point; or at least, I may have enough names but suspect I can do better. So I’ll do a quick Google Scholar search, limiting it to about the last 5 years, to find relevant papers that the authors haven’t  The authors of those papers can make good additions to my list.  Just as for the manuscript references, my search yields a mix of names I already know, and names I don’t.
  • My next step is an important one. I look at my scribble list, and when it includes names of senior people, I check their lab web sites in search of senior PhD students or postdocs.  My goal is to replace many or most of my senior names with more early-career ones.  This does three good things for me.  First, earlier-career folk are more likely to accept my invitation to review.  Second, earlier-career folks are more likely to give my reviewer list the diversity (of all sorts) that I’d like it to have.  And third: I’m utterly convinced that earlier-career folk provide (on average) substantially better reviews.
  • Some readers will be surprised that I’ve not yet mentioned DiversifyEEB. It’s a great resource (and there are beginning to be analogues in other fields). I use it from time to time, when I’m not satisfied with the diversity of the list I’m developing.  However, the keyword matching it provides is relatively coarse (compared to the matches I can get with Google Scholar, etc.), so I don’t find it terribly efficient as a source of reviewer names.  I think DiversifyEEB is a better tool for things like seminar invitations, grant-review panels, and award nominations than it is for identifying peer reviewers.  (It occurs to me now, writing this, that it might be a great tool for identifying up-and-coming candidates as new associate editors – where the coarse keyword matching is an asset, not a liability.)
  • Finally, I’ll check to make sure none of the reviewers appear in the paper’s Acknowledgements, or have other obvious overlap with the authors (same institution, recent copublication, etc.) The journal office will do the same.  Every now and then something like that will slip through, and normally the prospective reviewer will alert us to the conflict of interest, but I prefer not to have that happen.

Almost always, these steps will give me a 12(ish)-name list from which I’ll pick my top 6 – usually tilted towards early-career folk, and always better balanced (along multiple axes) than it would have been if I’d relied on author suggestions plus my own recollection.  There – that’s the curtain drawn back.

Readers who are also associate editors – which of your steps differ from mine?  (Fair warning: I will totally steal your best ideas!)

© Stephen Heard  December 20, 2018


*^Neither journal’s office has seen this post, and what I’m describing is my own workflow, not any kind of journal policy.

**^My sample size is still too small to describe numbers at FACETS.  Having to identify 6 (ish) to get two is actually a very good success rate; editors for other journals sometimes report churning through 20 or more.  I suspect this reflects the good reputation the Naturalist holds in the community, as a society journal that publishes high-quality work.

***^If you find you’re consistently putting more than one or two people on your “opposed” list, I’d suggest you might need to think about either how you, or your coauthors, treat other people in your field, or how you think about their motivations.  Or both.

Advertisements

17 thoughts on “How (as an editor) I choose lists of reviewers

  1. Pavel Dodonov

    Never again will I complain that a paper is taking too long to be sent out for review 🙂 Well, unless it is taking too long. How long would you say an editor needs to prepare this list, under normal circunstances?

    Like

    Reply
  2. Achilleas Kostoulas

    That’s a really helpful post; thanks for sharing your ideas! One more thing I like to look at is how often we’ve called on someone’s help. This is something of a problem, because some reviewers are more eager and better than others, and there is a tendency of calling on their help more often than others, which is rather unfair.

    One more resource I’ve found to be quite helpful for broadening my pool of potential reviewers is Publons. People listed there seem quite keen to review, most of the times, and I’ve managed to recruit a few very solid reviewers from there, whom I wouldn’t have found otherwise.

    Like

    Reply
  3. Chris Mebane

    Wow. You just give the Am Nat editorial a list and they do all the inviting and bird dogging? The AE doesn’t have to do it all? What a nice editorial office! I follow that approach almost exactly, except I send the first wave of 4-5 invites at once, instead of sequentially. If I invited 5 and get 5, then the author gets 5 reviews (hasn’t happened yet). Otherwise, you send a couple invitations, wait several days, send a reminder, rinse and repeat, give up, move down the list. It can take weeks just to get through the invitation list.
    One thing that’s easy to overlook is the work and interaction of the profession (i.e., paid) editorial office staff and the volunteer AEs and reviewers. I don’t think many (any?) journals would work well with all volunteer staffs. That’s one of the reasons peer review isn’t free and articles aren’t cheap. Some pretty good reading this week over at DE on costs of open access and at Scholarly Kitchen on why society journals such as Am Nat are different and worth preserving in the push to consolidate and reduce publication costs.

    https://dynamicecology.wordpress.com/2018/12/19/open-access-and-the-pain-of-switching-adaptive-peaks/

    https://scholarlykitchen.sspnet.org/2018/12/06/why-society-and-not-for-profit-journals-are-worth-preserving-better-economic-and-continuing-value-for-the-community/

    Like

    Reply
    1. Trish of the Naturalist

      Thanks! and yes, we can’t possibly afford to provide the support to peer review and publishing we do now if we are forced to flip to open access. I’m hoping diversity in journals can survive and there’s a niche for us (not for profit but not for loss). We want the editorial board to focus on content and not process.

      And yes, we do the finding and the bird dogging. We also do conflict of interest checks, we make sure reviewers are not asked too often, and we enforce a number of other rules to make sure the process is as fair and balanced as it can be. And we remind everyone that diversity is important. We also read through in-coming reviews for completion and tone–and we’ll work with reviewers if there’s an issue before passing it along.

      Like

      Reply
  4. sleather2012

    Interesting – I usually pick six to begin with, usually one from the author’s choice list and then use the journal search function, looking for any ECRs on it, then resort to Web of Science and Google Scholar, and I usually put an “old” friend on the list. I also, like you, use the reference section of the paper. I usually invite three at the same time; in my experience that will still probably only get me one accept to review 😦

    Like

    Reply
  5. chrissiepainting

    This is a really helpful post, thanks Stephen! I’ve been editing for a NZ journal for a couple of years now and sort of run through the same process as you, but perhaps not as thoroughly. This has helped to get a better process. Definitely highlights the importance of not only ECRs having good web presence themselves, but also encouraging their lab heads to ensure they have a list of postdocs/students on their sites too!

    Like

    Reply
  6. Pingback: Recommended reads #142: back to work edition | Small Pond Science

  7. Pingback: Did the other reviewer notice things you didn’t? That doesn’t mean you did a bad job. | Dynamic Ecology

  8. Marie E. McVeigh

    Your overall care and thoroughness in the process implies this, but I would have liked an explicit statement that, when using the author’s recommended reviewers, you validate the contact info. My understanding of the recent, significant abuses of the recommended reviewers list, it’s usually through an editor accepting and contacting the reviewer without that additional validation.

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      That’s an interesting point. Generally, it’s via the journal’s database; when it’s not, it’s my own search for the reviewer’s contact info (because I’ll check them out and send an email to the journal office recommending we add them to the database). So yes, we do validate the contact info. It’s distressing that one needs to even think about this – but one does, I suppose…

      Liked by 1 person

      Reply
  9. Pingback: How (as an editor) I choose lists of reviewers – Environmental News Bits

  10. Reanna Field

    I have a question about editors.

    Are they supposed to cross out things they don’t like, or is that unethical? I’m letting my sister read and annotate the third draft of my story, and she keeps crossing out what she doesn’t like and replacing words when she SHOULD be giving suggestions.

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      Well, that’s a different sense of “editor” – sounds to me that your sister is closer to what I’d call a “friendly reviewer” (someone you know but who agrees to read and makes suggestions). And that’s what she’s doing – crossing out one word and replacing it IS a suggestion. As the author you can make that suggested change, make a different change to resolve why your reviewer was dissatisfied, or you can keep the original – but always think carefully before taking that third option!

      Like

      Reply
  11. Pingback: From submission to publication - ESLR

Leave a Reply to Trish of the Naturalist Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.