Opinion, evidence, and preprints

Perhaps you’ve noticed that scientists, like other humans, can hold very strong opinions about certain things.* Perhaps you’ve also noticed that those opinions are sometimes backed up by voluminous evidence (gravity points down; climate change is real and caused by humans; vaccines are safe and effective) – but that sometimes they are not. Here’s a great example related to preprints.

Preprints are probably the most interesting development in scientific publishing in the last 100 years.** They aren’t interesting conceptually: the idea of making an un-peer-reviewed manuscript available to others before publication isn’t novel. But they’re interesting in their widespread adoption and distribution, and they’re especially interesting in the way people think their widespread adoption will change science.

See how I wrote that last clause? This is, after all, a post about opinion (and evidence).

There are many claims about the way preprints will change science: they make distribution of results faster; they allow clearer staking-out of priority; they make everything open-access; they let authors get the benefit of peer comments before peer review; they draw more attention to papers and thus increase impact (as measured by citation rate or altmetrics). I’ve been working on a second edition of The Scientist’s Guide to Writing, and one of the additions*** is a new chapter about journals and preprints. This called for some digging into those claims, so I could give some evidence-based guidance to readers considering the question of whether or not to post a preprint.

My digging was…interesting. I’ll focus here on the last claim in the list above: that preprinting increases citation rates. That seems like an important thing for authors weighing advantages and disadvantages of preprinting, and it’s a claim I see trumpeted quite a lot. So what does the literature say?

This won’t be a comprehensive review – that’s what the literature is for. But I can summarize fairly simply, and I’ll mention a couple of papers below the post.

First, if you compare citation rates, it’s clear that papers that are first posted as preprints are cited more than those that are not. It’s a big effect (anywhere from 30% to 100% more) and it shows up across preprint servers and across fields. This fact has been repeated a lot, usually with breathless excitement. What an easy way to increase your work’s impact, right?

And there are plausible reasons this might happen: preprints are open access, they launch faster than a paper, and they mean a piece of work launches twice rather than once. But what if you dig a little deeper? What if you read the papers reporting this effect a little more carefully?

Turns out the papers (but not necessarily the breathless reporting on them) explain that there are several reasons that preprinted papers might be cited more. First, the preprinted papers appear earlier, and can accumulate citations earlier in their publication history. Second, people who post preprints may be more likely to publish open-access (which also is associated with increased citation rates, after correcting for journal reputation). And third, and possibly most important: there’s some evidence that people are likely to post preprints of what they see as their exciting, important work, but less likely to post preprints of their more ordinary bread-and-butter papers. If they do, it’s hardly surprising if the preprinted ones are later cited more (granting, mind you, that an author’s idea of what’s exciting may not line up with readers’). So at least some, and possibly all, of the preprint-citation effect is not causal.

To be clear: none of this means you shouldn’t post preprints (I have), and it doesn’t mean preprinting isn’t helpful (maybe for citation, and definitely in other ways).  But I was surprised to find out that the ground under the much-ballyhooed preprint-citation effect is decidedly squishy.

I probably shouldn’t have been surprised. As scientist we’re supposed to be dispassionately data-driven and critical all the time, but we’re human. We have ideals we get excited about, we suffer from confirmation bias, and all those other messy things that go along with being human. It’s amazingly common for scientists to hold strong opinions about things they don’t have evidence for (cough, cough, two-spaces-after-a-full-stop-comic-sans-oxford-comma-serif-fonts-indenting-paragraphs just to pick a few from typography and punctuation). It’s always struck me as strange, too, that we make very basic errors in inference (such as not looking for obvious confounds) when thinking about things like preprints and spaces after a full stop – errors we would never tolerate in reasoning about our own study systems. (I hope.)

So are preprints an exciting experiment in scientific communication? Absolutely.**** Should you post preprints? Maybe – it’s up to you; there are plusses and minuses and they won’t balance out the same way for everyone. Should you post preprints and expect increased citation as a result? Probably not. Should you be cautious about breathlessly excited claims about advances in just about anything? Definitely.

People are interesting. Maybe I should have been a psychologist.

© Stephen Heard  December 15, 2020

Image: Various preprint server logos. Paleorxiv has, objectively, the best one. No, you don’t get to argue about that. It’s one of those strong opinions without evidence I’m talking about – but mine are right.

A couple of papers (and one preprint) that provide a good entry to this literature:

Davis PM, Fromerth MJ (2007) Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Scientometrics 71:203-215

Fraser N, Momeni F, Mayr P, Peters I (2020) The relationship between bioRxiv preprints, citations and altmetrics. Quantitative Science Studies 1:618-638

Schwartz GJ, Kennicutt RC Jr (2004) Demographic and citation trends in Astrophysical Journal papers and preprints. arXiv:astro-ph/0411275


*^Notice the Oxford comma in this post’s title? What about the single space after each full stop in this post? (I’d add the second post if I could, but WordPress strips them out. Yes, I’m a two-space person. Fight me.)

**^Yes, that’s a large claim. Yes, I’m making it without providing any evidence. Yes, I’m aware of the irony.

***^Don’t get too excited. It’ll be a while – some time in 2022, most likely.

****^Given that they’re 30 years old in physics, at least, my use of “experiment” is arguable.

 

3 thoughts on “Opinion, evidence, and preprints

  1. Jeremy Fox

    “It’s always struck me as strange, too, that we make very basic errors in inference (such as not looking for obvious confounds) when thinking about things like preprints and spaces after a full stop – errors we would never tolerate in reasoning about our own study systems. (I hope.)”

    This has always struck me too. But is it true? Is it the case that scientists who are (say) enthusiastic about “open science” make more/bigger inferential errors when reading studies of open science than they do in their own scientific work, or when reviewing the scientific work of others? I wonder…

    Like

    Reply
  2. John Kay

    Thanks for the thoughtful and dispassionate assessment of preprints. I find the discourse around preprints incredibly frustrating. Preprint advocates act as if they’re going to completely solve all problems with academic publishing – they’ll end publisher profiteering, free us from journal impact factors, and somehow make us all open science practitioners.

    But they don’t do any of these things. The big preprints end up in Cell, Nature and Science. Preprints have been around for decades and have killed roughly zero journals. Posted preprints hardly ever involve open data or preregistration. Most do not receive any public comments. They’re just early copies of manuscripts that usually end up published in a journal. To me, posting a preprint is like a conference presentation- usually worth doing but nothing life changing or revolutionary.

    Like

    Reply
  3. Pingback: Friday links: regrets about quitting academia, and more | Dynamic Ecology

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.