The shutdown of peer-review.io and the dynamics of volunteerism

Last week, Daniel Bingham announced that peer-review.io, the system he built for crowdsourcing peer review of academic manuscripts, will be shutting down (it seems to be already offline). It was an interesting experiment, and I’m disappointed, but I am also completely, utterly, 157% unsurprised. Here’s why.

Peer review is a form of academic service. For some folks it’s strictly a volunteer activity; for others it’s part of a paid job, but even then that connection is usually nebulous enough that taking on any particular review feels a lot like volunteerism.* Actually, a lot of academic service is like that: you may or may not be paid for it, but even if you are, the way that payment works is sufficiently nontransactional that it might as well be volunteer.** And that means academic service has dynamics that in my experience are pretty universal in the world of volunteerism. I think those dynamics explain pretty much entirely why peer-review.io couldn’t make a go of it.

I’ve been involved in a LOT of service, both volunteer and nebulously-paid, both academic and non-academic. And I’ve been involved both as the person doing the service, and as the person trying to recruit others to do the service. As a few examples: I’ve peer-reviewed hundreds of manuscripts, grants, and so on; and I’ve been a journal subject editor who invites those reviews. I’ve been Department Chair, and Acting Dean, and I’ve served on search committees for both roles. I’ve been a Council member for an academic society (the Canadian Society for Ecology and Evolution), and I’ve been its President and I’ve been the one recruiting candidates for its elections. I’ve been a Board member for our local Botanic Garden, and I’m currently its President and chair of its Board Development Committee. I could go on (can you tell I’m old?), but I’ll spare you.

There’s a pattern I’ve seen in virtually every one of these volunteer/service contexts, and it goes like this. We need people to volunteer, so those who are already volunteering reach out to folks they know who might have relevant skills and might be interested, and they talk them into volunteering too. (In a peer-review context, I’ve just described editors contacting potential reviewers; in a nonprofit Board context, I’ve just described existing Board members reaching out to friends and colleagues). Next, someone suggests that in fact we shouldn’t do it that way. After all, relying on our existing networks can be exclusionary (even if inadvertently so); and the system we have to build to institutionalize recruiting volunteers (like a journal!) seems like an inefficient use of resources. Instead, they propose, we should simply advertise the volunteer opportunity (=volunteer need) widely, and let people come to us. So we try this – but it doesn’t (usually) work.

It turns out, I’ve learned, that there isn’t a tumultuous crowd of people desperately seeking volunteer/service roles to take on. That doesn’t mean most folks are shirkers. Some are, to be sure; but most folks are just busy and know that there are a LOT of ways they could invest the volunteer time they have to offer. Here’s how Daniel summarizes this for peer review:

[The work of an editorial team includes] identifying and recruiting reviewers for papers… and convincing them to do the review. This is a huge piece of the labor of editorial teams.

Exactly! An Editor rarely spends much time fending off dozens of people clamouring to review a manuscript; instead, they may send six or ten or twenty invitations hoping – sometime begging – to secure two reviews.*** Search Committee members for a Chair positions rarely spend much time trying to decide between the six candidates vying for the job; instead, they spend their time arm-twisting their colleagues to find one good candidate. Board Development Committee members rarely spend much time adjudicating the competition for who gets to claim that prized open Board seat; instead, they spend their time beating the bushes for someone excellent and then convincing them to serve.

So: we can, and we should, think about models for volunteer/service recruitment that tap a wider array of people, or that are less vulnerable to the homogeneity that comes with recruitment via pre-existing networks. But we probably shouldn’t be surprised that this is a lot more work than just advertising a volunteer opportunity and waiting for hands to go up.

I don’t mean to disparage or poke fun at peer-review.io; it was worth trying, and it would have been lovely if it had worked (and Daniel’s post about its end is thoughtful and well worth reading). And I don’t mean to disparage or poke fun at those who push for “put it out there and see who offers” strategies; I always hope they’ll work, and it’s an idea motivated by faith in the goodness of people. But what stopped peer-review.io from working, and why similar “put it out there and see who offers” strategies usually fail, is (I think) just a general feature of the way our human communities are. People respond to people, not to abstract “opportunities”, and volunteer/service energy isn’t infinite. A lot of the systems we have – including journals – are designed to work from that fact.

© Stephen Heard  February 20, 2024

Image: volunteer balloons CC0 via rawpixel.net

---

*^Someone, either in the comments here or on social media, will highjack this post to make an irrelevant (to this post, I mean) and, by now, rather tiresome argument that academic publishers are evil and that our journal peer-review model takes advantage of unpaid labour. I don’t completely agree with that argument, but it isn’t necessarily wrong – but an argument that’s correct can still be tiresome and (in a particular context) irrelevant. Anyway, someone will do it; let’s see if they acknowledge, in doing it, that they’ve read this footnote. 😊

**^In other words: sitting on some departmental committees may well be part of the expectations of your paid faculty job; but there isn’t Committee Bonus Pay for those who sit on more, or more important ones; and there isn’t a particular amount of salary that you give up if you shirk the expectation. Being department Chair may well come with some extra salary and resources (it did for me), but that additional salary is rarely enough to make it “worthwhile” doing the job except as an act of volunteerism (it wasn’t for me). And so on.

***^The experience of preprint servers brings us to the same place. Preprinting is great – but the frequent argument that it’s great because it lets work get widely reviewed before it goes to a journal just doesn’t hold water. Most preprints don’t attract peer comments; and those that do are likely to be a biased subset, on the most controversial topics or by the highest-profile authors. An underappreciated role of journals is to be an attention-leveling mechanism, ensuring that every manuscript gets at least a couple of peer reviews.