Warning: header image captures this post pretty well.
Should peer review be open and transparent? Sounds appealing, doesn’t it? Who’d want to go on record as saying anything shouldn’t be made more open and transparent? Well, I’ll give it a go, because I’ve recently declined to review two manuscripts that looked interesting, for a reason that’s entirely new to me.* In both cases, the journals specified that by agreeing to review, I was consenting for my reviewer comments, and the authors’ response, to be published as a supplementary file with the paper. Sorry – I’m not having any part of that.
I guess I should start here: it’s not that I want the freedom to be nasty and unfair in private, anonymous reviews. Since I sign almost all of my reviews, if I need a reason not to be an ass while reviewing, I already have it.
So why is it that I’m willing for authors and editors to see my (signed) review, but unwilling to have other scientists see it (even unsigned)? In short, I can’t figure out what problem it solves, and I can think of new problems it introduces. And I have the same (negative) reaction to it as a reviewer, as an author, and as a reader.
First, as a reviewer. One major theme in my guidebook for scientific writers is that how you write a document depends on its audience – who they are, what they already know, and what you want them to learn. When I write a review, my audience is the authors and (to a lesser extent) the editor. That means I’m writing for an audience with lots of knowledge about the MS and its subject, and what I want them to learn is how to make it better. Adding in readers of the published paper as an audience changes everything! If I write the same review I normally would, those readers won’t understand it easily, if at all. (As a trivial example, consider the part where I write “what you say at line 173 seems to contradict line 67” – but you’ll be able to think of less trivial examples without much trouble.) Of course, I could write my review as an entirely different document that gives the “readers of the paper” audience what they need. However, that would be a much bigger job, and if I consent to do that, then I’ll agree to review only about 1/3 of the manuscripts I currently do. That’s a problem, of course – not because I’m personally indispensable (I’m not!), but because our publishing system is already drawing the available well of reviewers pretty close to dry. And if I put in that extra effort, what improvement in the progress of science will result? What evidence is there that readers of papers are hungering to read yet more supplementary material? What bad things have been happening in science that will be fixed by a handful of readers (and I suspect ‘handful’ is rather generous) being able to follow up their reading of a paper with readings of the original submitted version, then of the peer reviews, then of the authors response? And if these kinds of archives are really important, why don’t we ask for, and publish, every version of the paper as it goes through self- and friendly revision too? And every version that was previously submitted and revised at other journals?**
Second, as an author. If the reviews are published and so is my Response, the same multiple-audiences issue rears its head. I’ll have a choice: either write the Response as usual for editors and reviewers, and have it not be very useful to broader readers; or write for its new audience at the cost of effort I’d rather put into another paper. Plus, I’m grateful for the way reviewers help me find and improve weaknesses in my paper. Why would I want readers to see the pre-improvement version? If you’re like me, you can think of an embarrassing blunder or two a reviewer has saved you from. Thanks, I’ll keep my trash bin out of my Zoom background….
Third, as a reader. Honestly, as a reader I don’t care – or rather, I don’t care directly, or much. I can’t imagine a case where I’m likely to dig into the supplements to read the reviews, and the response, and the original version – except perhaps to indulge in a little shameful schadenfreude. But I am wary of the trend to more and more and more supplemental material. It inevitably eithers consume more of our limited publishing resources, or dilutes those resources (including attention paid by authors, editors, typsetters, and so on.). Without conviction that it makes my overall reading experience any better, I’m left assuming that it will make it, in some admittedly fuzzy and modest way, worse.
Because I sort of enjoy arguing with myself, let me close with my best case in favour of open peer review. That case is that unlike most of the other writing forms we practice, peer review comes without an easily available corpus of examples. I can tell my students to find examples of papers they find useful and readable, and to emulate them. I can’t tell them that about peer reviews (or the Response to Reviews). Open peer review would, I admit, solve that problem – although it’s peculiar that I’ve never once hear an advocate offer this as an argument. Perhaps they realize that although this would solve the problem, it would be a spectacularly inefficient way to solve it.***
It’s your chance now: use the Replies to convince me I’m wrong. Until someone succeeds, though, journals that require reviewers to consent to publication of reviews won’t be getting reviews from me. Which is OK, really – there are a ton of journals after me (just as there are after you, I’m sure). Filling my available reviewing hours isn’t ever going to be a problem.
© Stephen Heard June 29, 2021
*^I decline a lot more because the journals request unreasonably fast turnaround from their reviewers. As you’ll know if you’ve been hanging around here, I routinely refuse any review request with a deadline shorter than three weeks.
**^Some of you may (correctly) diagnose this last bit as a slippery-slope argument. Oddly, labeling an argument “slippery slope” is widely taken as a reason to disregard it. Really, though, slippery-slope arguments are some of the most important ones. That’s because the slipperiness of the slope indicates that it isn’t easy to come up with a defensible answer. Some of our most important decisions involve how and where to stop near, or on, a slippery slope.
***^There are, by the way, some excellent guides to writing peer reviews. There’s some coverage in The Scientist’s Guide to Writing, but admittedly not much. This guide from the British Ecological Society is particularly good.