We praise originality, but we don’t seem to value it

Image: Citation impact vs. originality, for 55 of my own publications.  See text for explanation.

 Warning: a bit cynical.

Last week I filled out a grad-school recommendation form for a terrific undergraduate student.  Among other things, it asked me to rate her “originality”.  That got me thinking.

We tell each other often that we admire scientists who are original thinkers. Originality is often an explicit criterion in manuscript assessment, in tenure assessment, even at science fairs.  The related idea of “novelty” is a major criterion in many (if not most) grant applications. Herman Melville might almost have been speaking for scientists when he said “It is better to fail in originality than to succeed in imitation*”.

So we praise originality. But do we value it?  I’m skeptical.  Science is a machine for discovering new things, and that seems intrinsically linked to originality – but as practiced, science often seems deeply conservative.  This isn’t a new insight; it’s a big part of Thomas Kuhn’s model of scientific progress.  But it occurred to me that I had at my fingertips an interesting way to assess how we actually value originality: the citation impact of my own scientific papers.  So I asked myself this question: are my most original papers also my most highly cited?  (I’m assuming here that citations are a plausible measure of how a field values a paper.  I’ll return to that assumption later.)

Quantifying citations is pretty easy.  I started with citation data from my Google Scholar profile, with a few minor edits**.  I regressed citation counts against years since publication, since older papers will inevitably have accumulated more citations.  The residuals from this regression are a measure of citation impact: a heavily cited paper will have a positive residual, and a poorly cited paper will have a negative one.  (This, by the way, is the same methodology I used to identify my most overcited and most undercited papers, but I’ve updated the data.)    I treated “empirical” papers (including observational, experimental, and theoretical work) separately from review papers throughout, as their citation impacts are quite different.

Quantifying originality is not quite as easy.  In an effort to limit the amount of procrastination involved in writing this post, I ranked my papers on an originality scale from 1 to 11, subjectively but as fairly as I could***.  A “1” is an utterly pedestrian paper, asking a familiar question and interpreting the data in familiar ways; perhaps it involves work in a different place or system from other studies, but it doesn’t break new intellectual ground.  An “11”, in contrast, asks a question nobody has asked before, or interprets data in a qualitatively different way from the previous literature.  (I’m most proud of my “11” papers, but I have a lot more “1” papers.)

Next I regressed citation impact on originality score. If we value originality, citation impact should increase with originality. It doesn’t:There is absolutely no hint here that we value originality!  If anything, it’s the reverse: although neither slope is significant, both are negative.  (Combining the two datasets makes the relationship nearly significant, P = 0.052, although I’m a bit worried about apples and oranges.)  My more original papers are cited less. The data are noisy, which isn’t a surprise – I haven’t tried to take out differences in journal profile, narrow vs. broad questions, theory vs. experiment, and so on.

I’m not surprised by this result, because it reinforces my preconceptions: that it’s easy to publish unsurprising, standard-approach work, and hard to publish anything outside the box****.  I realize this sounds like sour grapes (more about that soon), but I think it’s more than that. When you do something really original, it doesn’t fit with prior art in the field, and there isn’t a tidy way for people to cite it – even if they want to, and they may well not want to. Every field has a set of accepted ways to ask accepted questions. Papers that fit the mould slide through peer review and fit nicely into the citation lists of all the other papers asking the same accepted questions.  Take stream ecology, for example – there must be a bajillion papers putting leaves into litterbags and comparing decomposition rates among leaf species, water temperatures, flow rates, and so on.  (One of that bajillion is mine).  Or in plant ecology, there are a bajillion papers exposing seeds or seedlings to some kind of stress and measuring germination and growth (a couple of this bajillion are mine, too).  You’ll have your own favourite examples.

Now, precisely because the result reinforces my preconceptions, I’m cautious about it. Four reasons (and you may suggest more in the Replies):

  • I scored originality in my own papers, and because I remember at least vaguely which papers of mine get cited, I couldn’t blind myself. It’s possible that I’m just annoyed that any of my papers could have low citation rates, and that I’d rather think “Oh, that’s because my approach is ahead of its time” than “Well, I guess I wrote a lemon”.
  • Perhaps more original papers do have more influence, but it takes longer for that influence to be felt. Maybe original papers tend to be “Sleeping Beauties”, recognized and valued only long after publication. Mind you, as I get older, it gets harder to see this as plausible for my papers!
  • Perhaps more original papers are valued more, but that value is manifested in other ways than citations: inspiration of subsequent research, honours and awards, and so on. Although I haven’t been raking in those kinds of value markers either.
  • Perhaps citations are just a poor measure of value because they often serve other functions – they may be “throwaways” that check a citation box but don’t really constitute recognition or involve building on the cited study. Actually, I’m sure some citations are throwaways, although I think it’s hard to argue that citation counts don’t mean anything.

Or maybe Thomas Carlyle was right: “Originality is a thing we constantly clamour for, and constantly quarrel with”.

© Stephen Heard (sheard@unb.ca) May 10, 2017


*^At least, the internet says he said it.

**^I omitted 5 publications that are way outside my own field, because their citation data wouldn’t be remotely comparable.  These were three papers on sequence diversity in T-cell responses to viral epitopes, and two on the design of serine protease inhibitors.  I also omitted my 2016 and 2017 papers, which are too young to have a meaningful citation record.

***^If you don’t get the “11” reference, stop reading this post now and go watch This Is Spinal Tap. You’ll enjoy it far more than any blog post I could ever write.

****^I suppose I should acknowledge that I too have been guilty of casting suspicion on original thinking.

Advertisements

10 thoughts on “We praise originality, but we don’t seem to value it

        1. ScientistSeesSquirrel Post author

          Ah, sorry, I didn’t understand. Objectively, I can’t tell you (that’s what I thought you were getting at). Subjectively, the three papers I rated 11’s (two of the data points overlap in the figure) are:

          — Heard 2012 Use of Host-Plant Trait Space by Phytophagous Insects during Host-Associated Differentiation: The Gape-and-Pinch Model. International Journal of Ecology 2012:192345
          — Heard and Remer 2008 Travel costs, oviposition behaviour and the dynamics of insect–plant systems. Theoretical Ecology 1:179
          — Heard 2014 On whimsy, jokes, and beauty: can scientific writing be enjoyed? Ideas in Ecology and Evolution 7:62

          The third one is a weird paper and perhaps I should have dropped it. The other two are in lower-profile journals, and one could argue that’s why they’re low-citation; but in fact I’d argue the causality probably goes the other way; they’re in lower-profile journals because they’re 11’s. At least I’m quite sure that’s true of Heard and Remer 2008, which is my most-rejected paper and was passed over by 5 higher-profile journals.!

          Like

          Reply
  1. JC Cahill

    Steve – Thank you for the post. Its a great idea, and I actually tried to do a similar thing in the context of understanding the impact of ‘tenure’ (does it increase or decrease future creativity….), … but I believe there are some logical missteps, and perhaps a path forward.

    A very good reason one may feel their work is original is because they don’t know the literature. Thus, my ‘most’ original work from my perspective may also be my most pedestrian as viewed by more knowledgable folks. This would cause a negative originality vs citation relationship, as you observed. This is likely more pronounced as one bounces across fields … as our view of novelty may not be shared. Thus, i think the self-ranking is flawed.

    But, you could consider using a paper’s keywords as a metric. A more creative paper may perhaps have fewer routine keywords (e.g. diversity) than a more mundane paper. Thus, some measure of keyword isolation index at the time of publication may be a way to seeing how ‘out there’ the work was. Further, by recalculating isolation 5 years later (or some other amount), one could determine whether the field moved to embrace the ideas (reduced isolation), or not.

    just a thought…

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      Keyword isolation is an interesting idea! The main flaw might be that I’m terrible at keywords. (Also titles). Actually what I’d really like is to have somebody *else* score my papers for originality. Hmmm, who could I ask to do that? 🙂

      Like

      Reply
  2. Peter Apps

    What might be happening is that original papers tend to report smaller, less well planned studies that leave many questions unanswered. Big, well planned, studies need foundations, so they cannot be completely original, but they (should) provide clearer answers than the first “I wonder what would happen if” work that takes things in a new direction, and so they should get cited more.

    Like

    Reply
  3. Pingback: What’s in an idea? | Scientist Sees Squirrel

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s