The Delicate Dance of Peer Review

This is a guest post from Greg Crowther. Like a lot of us, Greg has thought about peer review from both sides of the table. It’s easy to get frustrated and proclaim that peer review is broken. It’s much more useful to come to a thoughtful take about what can be improved, and how. Read on!

Lately I’ve been thinking a lot about peer review. This is one of those academic topics that lend themselves to perennial hot takes like “peer review is broken.” My own not-so-hot take — broadly consistent with Steve’s perspective — is that the process is generally useful, often satisfying, and sometimes quite pleasant!

In a world where it’s hard to change anyone’s mind about anything, even (especially?) at faculty meetings, peer review can feel like a nice little oasis of rationality.

Advertisement

Peer review bias and double-blind reviewing: a new study

For all the strengths of our peer review system, there are reasons to worry that it has flaws too. One important worry is that peer reviewers (or editors) might be biased in their assessment of manuscripts based on their knowledge, or guesses, about the identities of the authors. Might manuscripts by female authors be treated more harshly? What about manuscripts by authors from the Global South? How might things change (and would it be for the better) if reviewers didn’t know who the authors were?

Preprints, peer review, and the eLife experiment

The “journal” eLife (more about the quotation marks shortly) made a splash last week, announcing a major change in their publication process. In a nutshell, eLife will no longer let peer review influence whether they accept or reject a manuscript. Instead, if they send it out for review at all, they’ll publish the manuscript along with its peer reviews. Authors can respond to peer review either by revising their manuscript or by writing a rejoinder – but they needn’t. You should read eLife’s rather breathless editorial (Eisen et al. 2022) to get the full picture.

It’s a major change for eLife, but I think it’s less revolutionary than it’s painted.

Let’s stop (usually) with the second round of review

I’m grumpy today about something that hasn’t even happened yet. Yes, that’s probably unreasonable; but I’m grumpy about something that happens too often, and I’m going to make myself feel better by venting just a little. I claim (at least partly because it’s true) that I have a real point to make.

Here’s what I’m grumpy about: second rounds of peer review.

Peer review, CVs, and what is Publons for?

Something a bit different today: this post is mostly just a link to a piece I’ve just published on jobs.ac.uk. There, I ask why early-career folks might get involved in peer reviewing, given that they aren’t paid to review (unlike many, if not most, more senior academics, for whom reviewing is part of the service component of the job). There are clear benefits to reviewing (which you can read about in the piece I linked to above*) but I don’t think one of them is giving you something you can list to good effect on your CV. Which raises the question: what is Publons for?

Should peer reviewers comment on writing style?

There seems to be a pretty widespread agreement that peer review should (even if it can’t always) identify flawed reasoning, improper statistical tests, and other serious issues with the inferences a manuscript makes. But should reviewers also make suggestions about writing style? About use of the active voice vs. the passive; about the use of contractions and other informality; about metaphors or even (gasp) humour? A lot of authors seem to think they shouldn’t, arguing that writing style is a personal decision that should be left up to a writer. Actually, I have some sympathy for that argument – the role of reviewers in crushing individual style is one reason that our literature lacks much individual voice, and pushback against beauty and humour is one reason it’s (mostly) so tedious. But in matters of style, should reviewers mind their own business?

Weird things scientists believe: that paying reviewers won’t cost us

Warning: a little ranty.

I’m fascinated by the weird things some scientists believe, in the face of what seems to me common sense and obvious constraints. There are many examples (like the common disdain for “nearly significant”), but the one I’ve chosen to offend people with today is a surprisingly common belief: that we could have journals pay their peer reviewers out of their profit margins without additional cost to authors. I see this claim frequently, most often on Twitter (although I’m not going to link to any particular exemplar, because the claim is too common to make it sensible to dunk on any one individual).

To get one thing out of the way immediately: I’m talking here about the notion that a journal could pay its reviewers.

Why I don’t want to be part of “open peer review”

Warning: header image captures this post pretty well.

Should peer review be open and transparent?  Sounds appealing, doesn’t it?  Who’d want to go on record as saying anything shouldn’t be made more open and transparent? Well, I’ll give it a go, because I’ve recently declined to review two manuscripts that looked interesting, for a reason that’s entirely new to me.* In both cases, the journals specified that by agreeing to review, I was consenting for my reviewer comments, and the authors’ response, to be published as a supplementary file with the paper. Sorry – I’m not having any part of that.

How to review an NSERC Discovery Grant

This is a guest post by Jeannette Whitton, Group Chair for Group 1503 (Ecology & Evolution) of the Natural Sciences and Engineering Research Council (Canada) (and Professor of Botany, University of British Columbia). She has extensive experience with the review and evaluation of NSERC Discovery Grants (among other things!) While Jeannette writes here specifically about reviewing proposals for Discovery Grants, much of her advice will serve you well in reviewing other kinds of grants, or grants for other agencies*. It will also serve you well in writing grants – because if you know what reviewers and evaluation panels are looking for, you can deliver just what they need.  Dig in!

Some weeks ago, you graciously agreed to review an NSERC Discovery Grant (DG) proposal, or possibly two or three**, which makes you an awesome person, especially in 2020. Because of confidentiality issues, we don’t get much training with reviewing grants – but just as for manuscript reviews, it takes time and care to provide a thoughtful grant review. How I review DGs changed after I served on the evaluation panel and got to see what was most useful, so I thought I would write down some thoughts about what to focus on. I hope this helps those who are new to NSERC DG reviews – or to reviewing grants more generally.  Comments are most welcome!

The list of disfavoured reviewers: who should be on yours? And will an editor heed it?

Last week, I wrote about lists of suggested reviewers (for manuscripts).  Most journals require them, although authors sometimes resent it; as an editor I use them and appreciate them very much..  But there’s another list that puzzles some authors: the list of disfavoured reviewers.  This is a list of people that you’re requesting not be asked to review your manuscript.  As an editor, how do I use that list?  And who (if anyone) should you put on yours?