Image: “Transparency”, CC BY-SA HonestReporting.com, flickr/freepress
Note: This is a modestly revised version of my original post, which was not written very clearly. (Yes, I’m aware of the irony.) It was easy, reading the original version, to think I was primarily objecting to journals publishing peer reviews. I’m ambivalent about that (and my arguments below apply only weakly to that situation). It should be clearer now that I’m focusing on authors publishing their peer reviews. If you’d like to see how my writing led folks astray, I’ve archived the original version here.
We hear a lot about making science more transparent, more open – and that’s a good thing. That doesn’t mean, though, that every way of making science more transparent should be adopted. It’s like everything else, really: each step we could take will have benefits and costs, and we can’t ignore real impediments. I worry that sometimes we lose sight of this.
One place I suspect we’re losing sight of it is in the movement for authors to publish their (received) peer reviews. (There are also journals that publish peer reviews, such as Nature Communications; I think this is a lot of work with dubious return on investment, but that’s a topic for another day). What I often see is the suggestion that whenever I publish a paper, I should post the full history of its peer reviews on Github or the equivalent. This lets readers see for themselves all that went into the making of the sausage. It’s worth reading a good argument in favour of this, and I’ll point you to Terry McGlynn’s, which I think puts the case as well as it can be put.
I don’t agree, though. Here’s why I won’t be posting my (received) peer reviews:
It probably isn’t legal. Peer reviews, like letters and other unpublished works, probably remain under the copyright of the writer*. Of course, the reviewer could agree to licensing terms allowing an author (or a journal) to post the reviews. I’ve never been asked to do that. If I’d signed a review (as I usually do), the author could contact me and ask permission to post it, but that’s never happened to me either. I think my experience is typical, which means that posting peer reviews will usually be a copyright violation.
- I’m not sure what serious problem it solves. Here are some possibilities (feel free to suggest others in the Replies). First, Nature Communications pitches their postings as a way to reassure people that peer review isn’t broken – but those who claim that it is, I’m afraid, will only cherry-pick posted reviews for evidence that seems to support their position. Second, it’s often claimed that the threat of posting reviews will make reviewers write more civilly. I get as angry as the next person when reviewers are rude or make ad hominem attacks, but editors should be redacting these and in my experience relatively few slip through. Third, posted reviews might shine a light on the shoddy if not fraudulent “review” processes at predatory journals, but this won’t be news to anyone who’s been paying attention – and the authors who post reviews won’t be those using, or being used by, the predatory journals. Fourth, perhaps reading the reviewer-author back-and-forth will help readers evaluate a paper’s claims critically. This seems to be a dubious claim given that commentaries and rebuttals seem to have little impact even when they’re published in journals. After all, in journals, at least commentaries have a good chance of being read, which brings me to my next bullet.
- It’s not clear who will read reviews if we post them. Post-publication peer review systems attract vanishingly few commenters, and this suggests that we really aren’t that interested in transcending the conventional view of the published paper as a one-way communication from author to reader. Maybe we’ll be more likely to read previous peer reviews than we are to post new ones (schadenfreude, anyone?), but I’d be surprised. The deluge of published papers is so huge that we’ve long since accepted we can’t read enough of them. Taking on the reading of posted peer reviews and rebuttals (frequently, combined, longer than the actual paper) will only make this situation much, much worse.
It will be a disincentive to review, and to review well. This is actually my most important objection. Peer reviewing is an important part of what makes academia work, but it’s voluntary and time-consuming and, as a result, editors often struggle to find willing reviewers. We’ll struggle more if potential reviewers are aware that reviews are commonly posted. Some people will simply decline, having no interest in being part – anonymously or not – of a back-and-forth that continues, publicly, beyond the acceptance or rejection of the paper. Others will stop signing reviews, being willing to reveal themselves to authors, but not to see their name posted for the world at large. I think the option of anonymity is important, but I’d prefer to see as many reviewers as possible decline that option. Still other people will move their more critical comments into the “comments in confidence for the editor”, making peer reviews less about improving manuscripts and more about gatekeeping. Finally, many scientists – like me – will still review, but we’ll see it as a different and more difficult job. When I write a review, I know I’m speaking to an editor and a few authors, and this lets me take some shortcuts in my text (because there are things I know the author knows). I can write colloquially. I can be candid about other papers the author might be citing, or not citing. Writing a review knowing it will be public changes all that**, making the task harder and more time-consuming, and making me likely to do less of it.
So, would posting our peer reviews make science more transparent? Well, yes, it would; but not that much, and with a set of unintended consequences that I’m sure extend beyond the ones I’ve flagged here. Call me unimaginative; call me conservative; call me a stick-in-the-mud – but I just don’t see a case for a dramatic overhaul of a peer-review system that really works quite well, almost all the time.
© Stephen Heard (email@example.com) November 1, 2016
Thanks to Terry McGlynn for comments on a draft of this post.
*^Copyright normally comes into existence upon creation of a work, not upon its publication or registration – at least in the US since 1976. (Important note: I’m not a copyright lawyer, and aspects of copyright law vary by country. US law seems best resolved and important to a large set of reviewers.) That means unpublished works have the same copyright protections as papers, books, and so on. This is very clearly true for letters, at least for the US and UK; see here for a clear US explainer. The same seems to be true for diaries and other unpublished written works. The extension to peer reviews doesn’t seem like a complete slam dunk, since one might be able to make some argument that a review is derivative of the paper it’s reviewing, or that it’s some kind of work for hire. I’m unaware of legal precedent on the specific peer-review issue, which is why I say publishing your peer reviews “probably” isn’t legal.
**^To be clear: I’m talking here about the kind of voice, and detail, and polish I’d put on a to-be-public review that I don’t normally put on a private review. I’m not endorsing reviewers who abuse the private nature of the process. Like most people, I claim that I won’t write anything in a review that I wouldn’t say to an author’s face. That’s not entirely true, actually – I sometimes don’t sign a review precisely because I wouldn’t be comfortable saying something to an author’s face. But this is because I fear the author’s bad behaviour, not because I’m worried about my own.