How (as an editor) I choose lists of reviewers

Image: The reviewer-selection screen at one journal I edit for.

Warning: more detail than you may care for. 

Every manuscript submitted to a (peer-reviewed) journal needs reviewers, and it’s the editor’s job to choose appropriate ones.  How does that happen?  Have you wondered? Well, I can’t tell you how it happens in general; but I can tell you how I do it.

I’m an associate editor for two journals* – The American Naturalist and FACETS.  Each time I’m assigned a manuscript to handle, I need to decide whether to send it out for review.  If the answer is “yes”, then I need to identify a list of possible reviewers, who will be invited, sequentially, to review until two (normally) accept.  For the Naturalist, I’m asked to make a list of 6 possible reviewers.  Usually we succeed in getting two of those 6 to review, but sometimes I need to identify more**. How do I choose names to include, on my first list or an expanded one?

I don’t have a complete rigid system, but here’s a typical workflow.

• To end up at a list of 6 reviewers, I’ll aim for a scribbled-on-paper list of 10-12. That’s partly because inevitably a couple of my brilliant ideas turns out to be unavailable for various reasons.  More importantly, it gives me some easy room to pick 6 of the 12 in a way that roughly optimizes a combination of fit and reviewer diversity – rather than being constrained to the first 6 I think of.
• My first move is to consult the authors’ list of suggested reviewers. Authors know their subfields well, and in my experience usually) make good suggestions.  I’ll almost always add 1-3 author suggestions to my list, although with the caveat that when we issue invitations from the list of 6, I want to use only one of the authors’ suggestions, and one of mine.  You might think I’m trying to prevent authors from gaming the system by suggesting only their friends.  My “1 of theirs, 1 of mine” rule does prevent that, but that’s not its primary purpose.  Instead, I’m aiming to represent perspectives the authors might not have thought of.  Those could be diversity perspectives – when authors name a bunch of senior people, those lists can be quite unrepresentative – or they could be disciplinary or organism-centric perspectives (the latter may be the most common thing I’m trying to broaden).  I think peer review is more useful if the reviewers are thinking from different places.

• I’ll also, of course, consult the authors’ list of “opposed” reviewers (at submission time, the authors can list people that they don’t want to review the MS). Almost all the time, I respect that list; there are enough wise reviewers in the world that it just isn’t a problem to do that.  I don’t guarantee to respect the list, though (most journals don’t).  And authors proposing more than a couple of opposed reviewers will earn a little bit of side-eye and a higher probability that I’ll overrule them – unless, of course, they provide a convincing explanation***.

• My next step is usually to add a few names that come to my own mind as suitable reviewers. When I began editing, I thought this would be my most important step; but I now understand that it’s my least  It’s rare that I let more than one of these “recollected” names crack my final 6.  However, doing this does get me thinking about the paper’s topic – and also other topics to which it might connect.  That helps me find other reviewers in my next steps.  This minimal use of recollected names is important.  It helps me minimize implicit biases, because like probably everyone else, I do a better job of representing diversity if I use recognition, not recall.  By “representing diversity” I mean diversity of all kinds: gender, career stage, academic/nonacademic affiliation, you name it.
• Next, I’ll add some names garnered from papers cited in the manuscript. I’m reading the manuscript anyway, at this stage, so it’s just a matter of noting citations that seem particularly key, or – and I almost always do this – skimming the Lit Cited section after I’m done for titles that are a propos.  Quite often this yields a couple of names that provoke a “gosh, why didn’t I think of them?”.  Almost always, it yields a few names I wouldn’t have thought of.
• I don’t usually have enough names at this point; or at least, I may have enough names but suspect I can do better. So I’ll do a quick Google Scholar search, limiting it to about the last 5 years, to find relevant papers that the authors haven’t  The authors of those papers can make good additions to my list.  Just as for the manuscript references, my search yields a mix of names I already know, and names I don’t.
• My next step is an important one. I look at my scribble list, and when it includes names of senior people, I check their lab web sites in search of senior PhD students or postdocs.  My goal is to replace many or most of my senior names with more early-career ones.  This does three good things for me.  First, earlier-career folk are more likely to accept my invitation to review.  Second, earlier-career folks are more likely to give my reviewer list the diversity (of all sorts) that I’d like it to have.  And third: I’m utterly convinced that earlier-career folk provide (on average) substantially better reviews.
• Some readers will be surprised that I’ve not yet mentioned DiversifyEEB. It’s a great resource (and there are beginning to be analogues in other fields). I use it from time to time, when I’m not satisfied with the diversity of the list I’m developing.  However, the keyword matching it provides is relatively coarse (compared to the matches I can get with Google Scholar, etc.), so I don’t find it terribly efficient as a source of reviewer names.  I think DiversifyEEB is a better tool for things like seminar invitations, grant-review panels, and award nominations than it is for identifying peer reviewers.  (It occurs to me now, writing this, that it might be a great tool for identifying up-and-coming candidates as new associate editors – where the coarse keyword matching is an asset, not a liability.)
• Finally, I’ll check to make sure none of the reviewers appear in the paper’s Acknowledgements, or have other obvious overlap with the authors (same institution, recent copublication, etc.) The journal office will do the same.  Every now and then something like that will slip through, and normally the prospective reviewer will alert us to the conflict of interest, but I prefer not to have that happen.

Almost always, these steps will give me a 12(ish)-name list from which I’ll pick my top 6 – usually tilted towards early-career folk, and always better balanced (along multiple axes) than it would have been if I’d relied on author suggestions plus my own recollection.  There – that’s the curtain drawn back.

Readers who are also associate editors – which of your steps differ from mine?  (Fair warning: I will totally steal your best ideas!)

© Stephen Heard  December 20, 2018

---

*^Neither journal’s office has seen this post, and what I’m describing is my own workflow, not any kind of journal policy.

**^My sample size is still too small to describe numbers at FACETS.  Having to identify 6 (ish) to get two is actually a very good success rate; editors for other journals sometimes report churning through 20 or more.  I suspect this reflects the good reputation the Naturalist holds in the community, as a society journal that publishes high-quality work.

***^If you find you’re consistently putting more than one or two people on your “opposed” list, I’d suggest you might need to think about either how you, or your coauthors, treat other people in your field, or how you think about their motivations.  Or both.

Advertisement