The one kind of review that really gets my goat

Photo: Journal of Universal Rejection coffee mug (crop), by Tilemahos Efthimiadis via, CC BY-SA 2.0.

Peer review gets a lot of grief.  It’s one of the things we love to say is “broken”.  It takes too long, or at least we think it does.  Occasionally a reviewer completely misses the point, goes on an ad hominem attack, or produces some other kind of idiotic review.  But for all the flak aimed its way, I’m convinced that peer review – overall – is fantastic; volunteer reviewers and editors have vastly improved nearly every one of my papers.

But there’s one kind of review that really burns my bacon. 

Maybe you’ve gotten this review too.  It’s the one that points out that there was a better way to do your experiment (or build your model, or make your observations).  Not, importantly, that the way you did it was wrong; just that there was a way to gather data (for example) that tested the hypothesis more directly, or were less noisy, or had better distributional properties.

Usually, I know there was a better way to do the experiment – after all, there almost always is.  Of course, that better way probably would have been more expensive, or more labour-intensive, or more time-consuming.  (Sometimes it would have required a smarter scientist, but let’s not think too hard about that.)   Very often, I tried that “better” way, and learned that it doesn’t work.  One or more of those reasons is generally why I’m not reporting the use of that “better” method in the first place.  Sometimes, to be sure, the reviewer’s better way really is better, and something I hadn’t thought of.  That’s exciting, because it puts a new arrow in my quiver for my next experiment; but it doesn’t have anything to do with judging the value of the data in hand.  So, there might have been a better way – so what?  I’d like to see my work judged not on whether there was a hypothetically better way to do it, but instead on whether what I actually did advances our knowledge of nature*.

As I’d love to write in a response letter (usually, but not always, I stop myself before actually doing such things):

You don’t have a choice between my data and perfect data.  You have a choice between my data and no data at all.

 I’ve had several papers rejected as a result of a there-was-a-better-way review**, and it makes me furious every time.  As a result, when I review, I try hard to assess the value of data in hand independently of my brilliant (and usually untested) alternative ideas.  If I’m tempted to write “it would be better to do X”, I think twice.  If I’m tempted to write “it would have been better to have done X”, I just don’t. Finally, as an editor, I try to recognize there-was-a-better-way reviews and reassure authors that they can disregard them.  Please join me.

© Stephen Heard ( September 27, 2016

*^By which I do not mean that my result is incontrovertibly, reproducibly “correct”.  Those who expect every scientific paper to be such have, I think, the rather naïve view that our literature should consist of a big pile of facts.  Instead, I mean that my result can add to the edifice we as scientists are collectively building, and in the long run in combination with other studies (both mine and yours) deepen or broaden our understanding.

**^None permanently. I’ve only ever submitted one manuscript that will remain forever unpublished – likely because (as reviewers argued, and I only much later came to agree) it was completely and irreparably wrong.


2 thoughts on “The one kind of review that really gets my goat

  1. Peter Apps

    Snap !, I’m responding to one of these at the moment, with the additional complication that the most interesting results were completely and utterly unexpected and so were not (and could not have been) taken into account in the original design.

    Liked by 1 person

  2. Michael James

    I’ve seen both sides of this issue. The situation you describe is more common, but occasionally I give a review recommending rejection because there is a better way. The test I have for this is that the better way has to be already known to many people (either because it has already been published or because it’s so simple that nobody bothered to write it down in their papers).

    Liked by 1 person


Comment on this post:

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.