Post-publication peer review and the problem of privilege

It’s been hard to escape calls lately for a paradigm shift in scientific publishing (most of them starting with a pronouncement that “publishing is broken”). We’re supposed to abandon pre-publication peer review, and replace it with a system of online preprint posting, open to anybody with no or minimal screening, that allows post-publication “peer review” in the form of a commenting forum. The preprint servers are here already: ArXiv has been an important channel for communication in physics and mathematics for years now, and BioRχiv is newly arrived in biology. What’s interesting is the other half of the prescription: the notion that preprint servers obviate the need for pre-publication peer review or for the existence of conventional scientific journals – and we’d be better off without them.

Does this make sense? Well, I’m on record saying that pre-publication peer review isn’t glacial and is nearly always helpful, so it won’t surprise you that I’m not enthusiastic about the preprint-with-post-publication-review model. There are many reasons why – today, privilege.

The packaging of papers into conventional journals, following pre-publication peer review, provides an important but under-recognized service: a signalling system that conveys information about quality and breath of relevance. I know, for instance, that I’ll be interested in almost any paper in The American Naturalist*. That the paper was judged (by peer reviewers and editors) suitable for that journal tells me two things: that it’s very good, and that it has broad implications beyond its particular topic (so I might want to read it even if it isn’t exactly in my own sub-sub-discipline). Take away that peer-review-provided signalling, and what’s left? A firehose of undifferentiated preprints, thousands of them, that are all equal candidates for my limited reading time (such that it exists). I can’t read them all (nobody can), so I have just two options: identify things to read by keyword alerts (which work only if very narrowly focused**), or identify them by author alerts. In other words, in the absence of other signals, I’ll read papers authored by people who I already know write interesting and important papers.

Who are these people I already know? By definition, they’re relatively senior – they’ve come to my attention because I’ve read their previous papers, which means they’ve published before (probably frequently). As a result, they tend to be authors who have privilege on two levels. First, they’re in established – often tenured – research positions, and usually have been for a while. Second (and statistically, if not causatively, correlated with this), they’re disproportionately (albeit not exclusively) people like me: middle-aged straight white men. And what about new scientists without long publication records – the younger and excitingly more diverse population of grad students, postdocs, and young faculty looking toward tenure decisions? They may be left on the outside looking in.

This seems to me a huge irony about proposals to replace pre-publication with post-publication peer review. At first glance, such proposals seem like the ultimate democratization: everyone’s manuscript on an equal footing. My manuscript and yours, a Nobel prize-winner’s and the rankest amateur’s, all available for readers whose comments will bubble the very best to the top. But this democratization will, I worry, turn out to be self-disrupting. Its very existence (coupled with the enormity of our literature) seems to force us to use prioritization signals that restore the very privilege we thought we were stamping out.

I don’t want this. My own early career benefitted enormously from the signalling associated with publication in traditional, pre-publication-reviewed journals. This paper, for instance, didn’t fit any pre-established line of research and came out when nobody had heard of me – but reviewers liked it, and it was widely read and cited because it appeared in Evolution. Only because it appeared in Evolution? Well, perhaps not; but I can’t imagine it having the impact that it did, if it had been cast adrift in a preprint server with only my undistinguished name to mark it. My paper would have caught in a catch-22: unread until marked by post-publication commenters, but uncommented until it was read. This is one thing our pre-publication system does very nicely: breaks the catch-22 by assigning “commenters” equally to every new manuscript entering the system, so that their quality can be signalled to readers without depending on authors’ prior fame.

So: should we upload preprints to the new servers? By all means. And we should welcome post-publication review, too – when it supplements, rather than replaces, the pre-publication review that can signal quality without depending on privilege.

UPDATE: Here’s a forcefully written and well-reasoned response from Micah Allen. I don’t agree entirely with his take on this, but you should read it!

© Stephen Heard (sheard@unb.ca) December 14, 2015

Related posts:

• Is publishing, and everything else, “broken”?
• Is peer review glacial, or do we expect too much?
• On the craziness and saintliness of peer reviewers
• When not to read the literature

---

*Full disclosure: I am an associate editor for The American Naturalist, and have been for 12 years. I’m also a subject editor for the new open-access journal FACETS. I have no involvement with any of the preprint systems.

**A problem that will have to wait for a future post, but in brief, I worry that this will lead to extreme contraction of silos within our broader fields.

***But aren’t these new scientists the very people pre-publication peer review is supposed to be biased against? It’s easy to find such claims, but data seem to suggest that privilege-bias isn’t a major problem. See here (including the comment thread), for instance, for discussion of gender and outcomes at several journals. For every reviewer who might give a famous person a free pass, there seems to be another eager to take them down. And experiments with double-blind review seem to suggest that while we can improve the peer-review system, the gains typically aren’t big enough to suggest we’d had really serious problems.

Advertisement