One reviewer is “concerned” that our hypothesis might not be true.
So were we. That’s why we did a study to *test* it.
— Stephen Heard (@StephenBHeard) September 17, 2017
Peer review is a dumpster fire, right? At least, that’s what I hear – and there’s a reason for that.
Last month, I got reviews back on my latest paper. Opening that particular email always makes me both excited and depressed, and this one ran true to form: a nicely complimentary opening from the editor and Reviewer 1 – followed by several pages of detailed critiques from Reviewer 2 – and Reviewer 3 – and, believe it or not, Reviewer 4.
As usual, there were comments I immediately agreed with, and comments I immediately took umbrage at. As I always do, I set the reviews aside for a couple of days so I could reconsider the latter kind. But when I pulled it back out and got to work, and as my “Response to Reviews” document* grew, I felt the umbrage rising again. And I gave in to temptation.
In particular, I gave in after reading a comment from Reviewer 3, who said they were “concerned” that our hypothesis might not hold. In a high dudgeon, I tweeted:
People liked this tweet. They really, really liked it. Seeing the Twitter “likes” accumulate, at first I felt validated. People agreed with me! That reviewer comment really was dumb!** But as the likes built up, along with replies and retweets using phrases like “the disaster that is peer review”, I started to feel a little guilty. I try to stay pretty positive on Twitter, and here I’d given into the urge to complain – and that complaint was rapidly turning into one of my most popular tweets ever.
But then it occurred to me that maybe the blame wasn’t entirely mine. I mean, I did the tweeting; but I didn’t do the liking. Why was this cranky tweet so popular?
Well, quite possibly, because people like crankiness, and they like thinking things are dumpster fires. This negative-news bias is a well-known phenomenon, in psychology and in business: bad news sells newspapers***. But I had inadvertently created an opportunity to measure the bias, as it applies to peer review. I just needed a control and a fair comparison. So after exactly 48 hours, I took a screenshot of my tweet’s likes and retweets. And exactly one week later, I tweeted admiringly about the same peer review:
I matched the timing (Sunday afternoons) and I did the best I could to match the original in structure and wording – leaving the only difference the tone. After the same 48-hour interval, here are my results: Bad-news tweet: 641 likes and 74 retweets. Good-news tweet: 122 likes, and just 7 retweets.
Twitter has spoken, and it isn’t pretty: we celebrate the idea that peer review is the ravings of idiots, and shrug at the suggestion that it might actually be helpful. (Granted, I have a sample size of two, but I’m an ecologist, and we rather like “two”. There’s a nice little paper in it for anyone who wants to pick up the torch and do this properly.)
So it’s hardly surprising that peer review gets a bad reputation. We give it that reputation by our selective attention to the times it gets things wrong. Of course it gets things wrong: it’s done by humans, and humans – all humans – are fallible. But it also, far more often, gets things right. We just don’t trumpet those from the rooftops. Who wants to stand at the water cooler telling everyone about how Reviewer 2 was so professional and polite and helpful?
So how do we counter the negative-news distortion of peer review (and everything else)? Well, we need to remind ourselves, frequently, that negative-news bias exists. We shouldn’t ignore bad news; bad news is real, and the world’s imperfections are important. But we also shouldn’t wallow in those imperfections, and we should recognize the good that exists alongside the bad. A lot of good, actually; far more than you hear about. And we ought to be countering our negative-news bias by consciously deciding to share good news stories just as much as we do bad – or perhaps, sharing them even more.
This matters. If we tell each other things are terrible, before long we’ll believe it – and not only that, we’ll have made it true. I’m sorry I was a part of that, and I’m here to call myself out. Yes, my reviewer said one dumb thing; but my paper will be greatly improved by their effort regardless. Why did I think it was so clever to shine my spotlight on the bad?
© Stephen Heard October 16, 2017
*^The “response to reviews” document is extremely important – and often mysterious to early-career writers. As an editor, I can tell you that a well-crafted response to reviews can have me mostly convinced to accept a revision even before I’ve read it. I cover the response to reviews at length in Chapter 24 of The Scientist’s Guide to Writing.
**^It is dumb. In fact, as an objection to a piece of science, it’s so jaw-droppingly idiotic that the reviewer obviously had to mean something else. I just couldn’t quite tell what, and I vented rather than working hard to figure it out.
***^For my readers under age 40: “newspapers” are called that because they were once printed on paper. And vendors “sold” them; or in other word, people paid money for them. How quaint! The world has changed – although bad news is still with us.