car_peer_review_comic_12

Are reviewers crazy? Or are they saints?

Image: © Jason McDermott, with permission

Are reviewers crazy? Or are they saints? Both, of course, and neither; but I’ll try to do better than that.

I’m writing this post because last week, the peer-review comic (above) was wildly popular on Twitter – it must have come up in my feed several dozen times. Which is reasonable enough, because it’s pretty funny, and we all feel this way about peer review from time to time. Those reviewers are crazy, eh? And sometimes they are. Sometimes they’re really crazy (for instance when they suggest a paper needs some male coauthors and bring the internet down in flames around PLoS One’s writhingly apologetic leadership)*.

But, here’s the funny thing. Everybody furiously retweeting that comic missed the first two sentences of the blog post it originally decorated:

 “I’m a big fan of peer review. Most of the revisions that reviewers suggest are very reasonable and sometimes really improve the manuscript…” [The Mad Scientist’s Confectioner’s Club, Jason McDermott]

I’m a big fan of peer review, too. As an author, peer reviewers help me improve my manuscripts so they’ll have more impact on more people. As an editor, they help me improve my judgment about which papers I should accept, and what changes will make them better. And they do this as volunteers, without any compensation from the journal or from the author, without much (if any) credit from their academic institutions, and often without much in the way of thanks, either. If sainthood is a bit of a reach, at least a healthy dollop of gratitude seems in order.

It’s easy to forget this, as an author, because peer reviewers are playing two roles at once: gatekeeping and manuscript improvement. As an author it’s only natural to focus on their gatekeeping role: to see the reviewers as what stands between you and publication. Maybe they’re the boot holding your career down; maybe they’re just forcing you to add a red-tinted windshield and a top-mounted laser cannon. This is an understandable way to think about peer reviews, but it isn’t the right way. It’s their manuscript-improvement role that makes peer reviewers invaluable. If you’re thinking about your reviews the right way, every comment is a chance to make your paper better.

Now, sometimes reviewers say dumb things. Of course they do – they’re human! They may overlook something in your manuscript; they may misunderstand something; they may lack knowledge you’d like to assume all your readers have; they may be wrong about a point of statistics; they may just be tired, or hung over, or having a bad day. We all do all of these things from time to time. But that doesn’t make the peer review system “broken” (as one often hears). In fact, even a reviewer comment that seems completely moronic has value. That’s because your reviewer is simulating the experience your reader is going to have, and the reviewer’s mistake (if that’s what it is) is alerting you to a (probable) future reader mistake. Ask yourself: how did the reviewer get that crazy idea? Are you sure it wasn’t because your paper was a little bit unclear, or a little bit disorganized, or a little bit too lengthy?** (And, for that matter, are you sure that red-tinted windshield doesn’t actually make your paper better?) Anything that helps tell your story more clearly to more readers (including busy, tired, or careless ones) can only further your contribution to science.

So, reviewers (sometimes) are crazy, and reviewers (sometimes) are saints. The question, of course, is how often they’re one and how often they’re the other. I’ve seen a lot of peer reviews in my career as an author and an editor – certainly at least a thousand. How many of those were perfect? Probably none. How many were crazy? I can think of two. And that seems pretty good to me, for a system that relies on imperfect humans donating their time and effort to perform a difficult and often thankless task.

I’ll be honest: from time to time I’ve resented my reviewers, cursed my reviewers, even made fun of my reviewers – because like them, I’m human. But I’ve also thanked my reviewers, privately and publicly, because they’ve made every one of my published papers better.

Every one of them.

Thank you.

© Stephen Heard (sheard@unb.ca) May 5, 2015

 Thanks to Jason McDermott for comments on a draft of this post.


*Of course, I didn’t see the whole review, nor the manuscript being reviewed, so maybe the comments only seemed crazy because they were taken out of context. Yeah, that’s it….

**This is something I really didn’t understand early in my career. I still don’t understand it when I first read through each new review I receive; but I now know to put the review aside for 24 hours, and with that perspective I almost always see a loose thread in my manuscript that could trip up a reader.

30 thoughts on “Are reviewers crazy? Or are they saints?

  1. Pingback: Causal language in ecology papers – Ecology is not a dirty word

  2. kerkhoffa

    Great post, Stephen. I too am fatigued with how the inevitable imperfections of peer-review cause many to jump to the conclusion that it is “broken.”

    Like

    Reply
  3. jasonmcdermott873738820

    Stephen- great post. As you mention, I support peer review and it’s clear that it works in a lot of cases. I really like your point about even off-target comments from reviewers being important to consider. It’s these kinds of comments that should make you as an author think that your communication has not been effective, and usually these are very informative as to where the breakdown occurred. Thanks for your post- I think this kind of discussion is great to have.

    Like

    Reply
    1. Jeremy Fox

      YES! Reviewers are us! I still don’t understand why so many people apparently don’t understand this. (Or maybe most people do understand it, and it’s just that the small minority who don’t are very vocal?)

      Relatedly, we all disagree with one another a fair bit about what science is worth doing and how best to do it. That won’t change if you get rid of peer review or radically reform it or whatever. People who complain about the peer review system often don’t seem to realize that what they’re really complaining about is people who disagree with them.

      https://dynamicecology.wordpress.com/2012/11/25/how-random-are-referee-decisions/

      Like

      Reply
  4. Massimo Meregalli

    A very interesting project for a new peerage system was quite recently developed. I have used it, and it proved to be very useful. Significant improvements were proposed to a paper of mine, and it ended in a much faster acceptance of the paper. https://www.peerageofscience.org/

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      There are a couple of such systems, and it will be interesting to see if they “stick” – but I’m not sure they have much to do with peoples’ complaints about crazy peer reviews. Do you think there’s any reason to think the quality of reviews will be higher or lower with such systems?

      Liked by 1 person

      Reply
      1. Massimo Meregalli

        Well, difficult to say. But a few points can be considered. First of all, this is a pre-submission review. The scientists that are in the panel of Peerage (there are more than one thousand now, since any author who has a paper revised by Peerage is added and should later revise a paper) receive an alert when an article that fits with their specialization is submitted to Peerage of science. They can decide whether doing the review or not – of course, usually not, but if they think it is interesting they can accept. In my case I had four reviews of my paper, and two were extremely precise and well done, with much constructive criticism. And when I did the review of an article, there were four other scientists doing the same. This is interesting, since the peers are chosen among many scientists, whereas an editor usually has a small batch of possible peers and always use them (I know… I make reviews for always the same journals). Then, there is a peer of peers, that is, after the procedure is completed each peer evaluates the other peers. So the author of the paper receives an evaluation of the various peers. An author can decide if it is OK to accept the review or not, and since the editor receives the whole history of the paper, from the first submission to the edited version after the suggestions, he can already see which were the peers’ comments and how the article was improved. In general, I think that this is a fast and good system, the whole procedure is completed in less than two months. And of course, an editor can directly accept the paper considering as already revised or, more usually, send it to another reviewer. But, at least in my experience, this system worked pretty well and it can diminish the burden of the editor when he looks for peers for the articles that are submitted. Indeed, there are several journals that now accept, more or less fully, this procedure. Of course, this system does not have much to do with people’s complaints, but if it becomes more widely accepted, also by editors, then the problem of some “crazy” review is overcome since the review is pre-submission.

        Liked by 1 person

        Reply
  5. Mason

    Very nice post. I have had great experiences with reviews but also very poor experiences. I think it’s unfair to say the entire system is broken, but some tweaks or fixes are needed. One poor experience I had with a MS was one positive and one negative review, with the negative reviewer’s comments slightly off base (insert borderline crazy), but the editor siding with the negative review. In this case it was revise and resubmit. This went on for three rounds of reviews with the same result all three times! The paper was finally accepted and the final version was closer to the original version because the last of the six reviewers comments contradicted the the first reviewers comments. It took 23 months for the paper to be finally accepted, which seems too long.

    I’d like to know why editors seem to side on the more critical or “crazy” review and either reject or give revise-resubmit instead of siding with the positive review?

    Like

    Reply
  6. JJackson

    My review of this post: Everyone appreciates good reviews, i.e., those which suggest useful improvements. The issue is reviews that are in some sense unprofessional. The post sweeps these under the rug by terming them “crazy.” More useful would have been to present a taxonomy of what constitutes unprofessional reviewer behavior. I’ll give two examples from my experience. 1) A negative reviewer of an investigation of continent-wide environmental and behavioral correlates of phenotypic variation in a large taxon failed to disclose that he had simultaneously submitted a brief note on a related subject to another journal. 2) A negative reviewer stated that the paper would be acceptable if he were included as an author because he, being an expert in a field in which I was an outsider, could improve the presentation. These are issues of ethics, as are doing a review in a hurry, in a bad mood, or hung over. Being a volunteer does obviate one’s ethical responsibilities.

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      Agreed – there are different kinds of unhelpful comments, and the two you describe are probably unethical (certainly the second one). The taxonomy you suggest would indeed be a useful post (although a quite different one) – thanks for the suggestion!

      Like

      Reply
  7. Pingback: Friday links: the rush to declare the Anthropocene, microaggression on campus, and more | Dynamic Ecology

  8. Pingback: The dumbest thing I ever said to a reviewer | Scientist Sees Squirrel

  9. Pingback: How I started blogging (reposted interview with Paige Brown Jarreau) | Scientist Sees Squirrel

  10. Pingback: Was Barbara Cartland a genius? And are you? | Scientist Sees Squirrel

  11. Pingback: How long should peer review take? | Scientist Sees Squirrel

  12. Pingback: Creativity, play, and science | Scientist Sees Squirrel

  13. Pingback: Is everything “broken”? | Scientist Sees Squirrel

  14. Pingback: Praise | Scientist Sees Squirrel

  15. puddleg58

    I’m going through this for the second time. The reviewer’s comments are mostly very helpful this time and the paper is better for them.
    However, and there has to be a however, sometimes it’s as if the editors don’t tell the reviewers about the word limit and reference limit when you’re submitting the kind of short article that has to meet these. The reviewer wants you to address a, b, and c, plus reference x, y, and z, and in an ideal world you would, but you’ve already used your word and reference limit and x, y, and z are in every textbook and not in dispute.
    Or, the reviewer wants you to write about their pet idea, which is different from the subject of your article, and which cannot co-exist with it, and in any case is not within your area of expertise. Basically the reviewer wants to use you as a ghost writer, when the proper thing to do would be to wait till you publish then send a letter to the editor using your article as a pretext to discuss their pet idea, assuming that the editor will be able to see the relevance, which is currently escaping you. The eventual solution is to contrive a paragraph explaining why the paper’s focus means that such an excellent idea cannot be discussed except in passing.
    Still you’re grateful because the reviewers have pointed to holes that would likely have sunk your paper had it in fact been published like that.
    And anyway, if you can survive writing with coauthors, reviewers (mostly) don’t seem so bad.

    Like

    Reply
  16. Pingback: Searching for squirrels | Scientist Sees Squirrel

  17. Pingback: Post-publication peer review and the problem of privilege | Scientist Sees Squirrel

  18. Pingback: The best Acknowledgements section ever | Scientist Sees Squirrel

  19. Pingback: Writing a book: what a long strange trip it’s been | Scientist Sees Squirrel

  20. Pingback: How to handle an idiotic review | Scientist Sees Squirrel

  21. Pingback: The one kind of review that really gets my goat | Scientist Sees Squirrel

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s