It’s been a while since I’ve been this proud of a paper

I don’t usually blog about my own papers, except in some rather meta ways, but last week saw the publication of a paper I’m really, really proud of.  And it has some interesting backstory, including its conception right here on Scientist Sees Squirrel.

The paper is called “Site-selection bias and apparent population declines in long-term studies”, and it’s just come out in Conservation Biology.  It started, back in August of 2016, with a post called Why Most Studied Populations Should Decline.  That post made a very simple point about population monitoring, long-term studies, and inferences about population decline.  That point: if ecologists tend to begin long-term studies in places where their study organisms are common (and there are lots of very good reasons why they might), then we should expect long-term studies to frequently show population declines simply as a statistical artifact.  That shouldn’t be controversial – it’s just a manifestation of regression to the mean – but it’s been almost entirely unaddressed in the literature.

A bunch of folks read that blog post.  Some were mortally offended. Some protested that it couldn’t possibly be true.  Others protested that it was obviously true, but wasn’t important.  Still others protested that it was true, and important, but everybody already knew about it (this third group was, it seems, blissfully unaware of the existence of the first two).  Finally, a few made excellent constructive comments about it, and suggested further work to turn a thought experiment (blog post) into some actual science (paper).

Among that last (constructive) group were two people who ended up coauthoring the published paper: the terrific Auriel Fournier (now Director of the Illinois Natural History Survey’s Forbes Biological Station), and the equally terrific Easton White (now a postdoc at the University of Vermont).  There were also several who didn’t end up as coauthors but played important roles (more about that below).  Auriel and Easton pointed out to me that we could do two important things to build on my small thought experiment.  First, we could use simulation studies to ask whether the effect is potentially important – whether, given simulated population trends  (or the lack of them) and particular patterns of density-biased initial site selection, analyses would overestimate declines enough to matter.  Second, we could use compilations of data from long-term population studies to ask whether there’s evidence of the site-selection effect compromising real studies.  The test takes advantage of a simple prediction: if trends are distorted because researchers initially choose sites with dense populations and avoid those with sparse ones, then trimming the first 10 years (“10” is an arbitrary choice) from the time series should reduce the frequency and magnitude of inferred declines.  With enough long-term population-dynamic datasets, we can test that prediction.

We did both of those things, and the answers are “yes” and “yes”: simulated data show that the problem can be a big one, and there’s strong evidence that the problem occurs in real studies.*  We also did one more thing.  We found what we believe to be the only high-profile paper that mentions the problem (Pechmann et al. 1991) and traced every paper that has ever cited it.  We discovered that of 478 citations, just one cites that paper for its mention of the site-selection problem in inference; over 40 (by comparison) cite it as having inferred population declines.  So: the effects of site-selection bias are real, and important, and largely undiscussed.

I mentioned that I’m pretty proud of this paper.  There are three major reasons.

First, I think the paper is important – perhaps my most important ever.  That’s not because I doubt the existence of population declines – many species are clearly in deep, deep trouble, we’re the cause of it, and we need to take action.**  It’s because we desperately need our inferences about population declines to be accurate.  If we’re to allocate conservation funding – and we must allocate it, as we simply can’t target all species equally – then we need to know which species are in the most trouble.  Our paper identifies an extremely important way that we could be wrong about that, and suggests ways to avoid the problem.

Second, I think the paper provides some evidence that I haven’t wasted the time I spend writing Scientist Sees Squirrel.  I do invest time in this blog (a few hours a week, usually, mostly at times when I’m not otherwise very productive; but still, it’s non-trivial).  So I’m proud to see that the ideas I write about here can mean something to my colleagues – even, at least once, can excite some sharp early-career folk enough to have them help me write a paper.  How cool is that?  (Mind you, Why Most Studied Populations Should Decline isn’t the only post I’m proud of.)

Third, the paper took some serious persistence to get published, and I’m proud that we were able to stick to the Tubthumping Strategy and see it through.  The paper was rejected, with remarkable vigour, by three journals in rapid succession – and in the process received some truly stupid comments from reviewers and editors.  Now, I’m a huge fan of the peer-review process in general, but that doesn’t mean all reviews are always wonderful. In this case, it was completely obvious that reviewers had made up their minds that they hated the manuscript and disagreed with its conclusions before they bothered to actually read it.  They accused us of not citing enough papers (when our point was that an issue was unaddressed in the literature), and of citing irrelevant ones, and of failing to locate the “many” papers they knew dealt with the issue – not one of which they could actually identify. They accused us of not understanding elementary population biology (I teach population biology) and basic statistics (I teach statistics).  They objected vociferously to our having said things that we actually didn’t say, and to our not having said things that we did, very clearly, say. I’ve never seen as clear a case of the Boxer Effect:

            A man [sic] hears what he wants to hear/And disregards the rest.***

Look, maybe in a way this was understandable: we were pointing out an awkward truth about mistakes a subdiscipline is making – and we were doing it from outside the subdiscipline in question.  But I hope I never, ever write the kind of peer review that we got, repeatedly.

We’re grateful to the reviewers and editor at Conservation Biology, the fourth journal we tried, who were willing to consider our responses to comments, and ultimately accepted the paper.  The lesson: if you continue tubthumping long enough, you can publish the good work you believe in.  Another lesson: even truly stupid reviewer comments – like those on our first three submissions – can be used to improve a paper.  The published version is unambiguously better than the first version that went out.

Now, there’s an important way I’m not proud of our new paper, and that’s that there are a couple of people who made important contributions early in the project, but who aren’t part of the final author list.  There were misunderstandings about intended authorship that left me disappointed and upset – with myself as much as anything (I should stress that these misunderstandings were in no way the fault of my earlier-career coauthors).  I’m not going to go into what happened here; it wouldn’t be fair to use this platform to tell just one side of the story.  But let me encourage everyone to do as I say, and – apparently – not as I do: get clear agreements on authorship at the start.  Critically: those clear agreements need to cover not just the ways in which collaborators join the project (that’s easy), but also ways in which one might recognize and adjudicate their having left the project (much harder).

Despite those regrets over coauthorship misunderstandings, and despite the worst peer review experience of my career, the project was enormous fun.  I was kept motivated by my awesome coauthors, by Twitter followers and blog readers, and by discussions with colleagues when I presented the work at conferences and in departmental seminars.  Thanks to every one of you!

Finally: my goal here at Scientist Sees Squirrel is definitely not to pump my own papers; but I’m going to close by suggesting that if you ever read on of my papers, make it this one.  It’s an easy read, we think (we worked hard on that), and we’d love to see its message spread widely.  Read it – even just skim it, if you like – and then pass the link on to a friend or colleague.  Thank you.

© Stephen Heard  June 25, 2019


*^We can’t, with our method, identify a site-selection problem in a particular dataset, but we can infer that a sizeable fraction of a set of datasets have site-selection problems.  What to know more?  Read the paper!

**^My deepest fear is that some right-wing, anti-science troll or buffoon will pick up on the study, and misread it to say that “oh, scientists have now declared that all this hand-wringing over endangered species was all based on a big dumb mistake”.  That’s not true and the paper doesn’t say that, but the world is not short on right-wing, anti-science trolls and buffoons who can’t or won’t understand that.

***^From The Boxer, lyrics by Paul Simon.  Who should have won the Nobel Prize in Literature that went to Bob Dylan – an opinion that is both utterly irrelevant to this post and utterly resistant to objective argument either for or against.

 

Advertisements

23 thoughts on “It’s been a while since I’ve been this proud of a paper

  1. Jay

    Congratulations to you and your coauthors for an important contribution, and one that is testimonial to just how messy science can be. Hopefully, there is robust follow-up to validate, refute, or extend your findings. That’s how science progresses.

    Like

    Reply
  2. sleather2012

    Interesting – the site of my 20-year 52 sycamore tree data set was picked without regard to initial density of the organisms that I studied, two of which I didn’t even know were there but there are population declines evident, but perhaps 20 years wasn’t long enough? Things may have recovered/changed over the last seven years that I haven’t been looking at them 🙂

    Like

    Reply
  3. Jeremy Fox

    Great posts, great paper.

    Question: if there are populations that we began monitoring because they were rare/declining (or thought to be rare/declining), those populations should subsequently appear to increase because of regression to the mean. Can you check for that possibility in your data as well?

    Liked by 1 person

    Reply
    1. Jeremy Fox

      I should add that I know you mentioned this possibility in your paper. Just wonder if you had any further thoughts on how to pursue this possibility that didn’t make it into your paper.

      Like

      Reply
        1. Jeremy Fox

          I guess the obvious data with which to check this out would be cases in which monitoring programs were initiated in response to concerns that a focal species was rare or had declined. Those data must exist for some species, right? Like, species listed under the Endangered Species Act in the US or SARA in Canada? Although there I guess you run into the problem that those species also are subject to management interventions intended to increase their abundances. Hmm.

          Like

          Reply
  4. Pingback: Statistical vignette of the day as a teaching tool (UPDATED occasionally) | Dynamic Ecology

  5. Brian McGill

    Great post, great idea. I’m looking forward to reading & citing your paper (I’ve already cited your prior blog post a couple of times).

    My adviser used to say its the really obvious ideas that people think we must already know them but that aren’t yet in the literature that are really important, but reviewers have a hard time seeing that. That’s the charitable interpretation of your review experience.

    The less charitable one, with some grounding in my own experience, is that there is a sizable minority of ecologists who think it is their duty to keep the message of unrelenting doom so clear and unquestioned that they will use any means possible to stand in the way of being scientifically careful and cautious.

    Personally I think the latter is a mistaken and overblown fear. Like you, the authors of the Dornelas 2014 paper on local richness averaging flat feared misuse by the anti-environmental movement. We only ever detected one obscure example of this (it went nowhere) and there was plenty of coverage of the larger issue of biodiversity trends that was reasonably accurate and nuanced. I’ll be curious to hear a month or two down the road what your experience is.

    Liked by 1 person

    Reply
  6. Terry McGlynn

    I also think it’s a helluva great and important paper.

    The initial post called to mind a Master’s student who I got to know from working at my field station in Costa Rica. This student was sent down to work on a system that their PI was familiar with from a few decades earlier. The student was reassured that the animal was super-duper common and quite right for a one or two right-sized field seasons for a MS thesis. But it turned out that this animal used to be common, but not longer was common. (I remember seeing it all the time in the mid-90s, but then over the 2000s, I saw them less and less.) I don’t know if its decline is a conservation biology problem, or if it just temporarily had a big burst in densities in the 80s and 90s, and it’s gone back to a perfectly typical and fine density. Or presumably it’s something more complex than either of those two scenarios. Anyhow, this person’s advisor locked in his mind that this was a Common Animal, and that things would just continue to be that way. Because it’s easier to study common thing, this bias can be so pervasive! It’s nice to have a paper to cite.

    If you’re wondering, this student eked by with just enough data for the MS thesis, though with too much work than should have been necessary, the paper came out. They recovered, picking long-term systems based on assumptions about future densities has a lot of embedded assumptions, eh!

    Liked by 1 person

    Reply
  7. Florian Hartig

    I have wondered previously about regression to the mean problems in this context, so it’s good to see this discussed in the literature.

    Skimming your paper, I have one doubt though: unless I misunderstand, you mainly demonstrate that regression to the mean effects are POSSIBLE (which seems self-evident to me), but not that they are ACTUALLY THERE. I doubt that it’s possible to detect these effects statistically by fitting curves to population decline rates, because this signal is possibly confounded with true declines. I guess the most direct evidence would be to show that initial measurements of sites / plots in important datasets have higher population sizes / diversity than random locations. Do you have any evidence to show that this is the case, or any other idea of how we could estimate the actual extent of the problem?

    Like

    Reply
      1. Florian Hartig

        Yes, I had read this, but that is what I meant by “possibly confounded with true declines”, i.e. true decline rates may have simply changed. I see you mention this shortly in the discussion, and dismiss it as an unlikely explanation, but I’m not sure, I guess it depends on the location and context of the study.

        I agree that the pattern is indicative of a regression-to-the-mean effect, but what was asking if we could somehow generate direct evidence for it. I guess somewhat more direct would be, for example, if surveys that have a proven random sampling design find lower decline rates than surveys with an undisclosed sampling design.

        Like

        Reply
        1. ScientistSeesSquirrel Post author

          That would be an interesting approach – I think a contrast between the Christmas Bird Count and the Great Backyard Bird Survey might be interesting? If I’m right about how those work, and I might not be.

          Like

          Reply
  8. Ulises Balza

    Congrats! Great paper and idea! And it is very encouraging to have examples of published good ideas, just that.

    Regarding to the authorship and based on my very short scientific expierience, people do not want to state places in papers in advance, looking you like ‘really? so are you interesting in publishing rather than science?’
    About the 4th journal thing, I’m sorry, but it is also very encouraging that these kind of things are not exclusive of junior scientists. I had my first good experience with Biological Conservation in a comment we published in May, but in general it is very sad to deal with those destructive comments…

    Finally, regarding to population biology, have you seen this comment on the paper on insect decline? https://www.sciencedirect.com/science/article/abs/pii/S0006320719303325

    Ulises from Ushuaia

    Like

    Reply
  9. Pingback: Friday links: regression to the mean vs. the purple-snouted crompus, profs vs. meetings, and more | Dynamic Ecology

  10. Jeremy Fox

    So, here’s an off the wall question for you Stephen: what if there’s no such thing as regression to the mean because populations don’t have well-defined means?

    That is, what if population dynamics are non-stationary, at least on sufficiently long timescales? Like an unbounded random walk, for instance, or some other unbounded stochastic process? So that, in a statistical sense, there’s no such thing as a “typical” density around which the population tends to fluctuate, or towards which it tends to revert in the long run.

    Just thinking out loud, I think one answer is that regression to the mean might well be a timescale-dependent phenomenon. Maybe populations don’t have a constant long-term mean density. But on some shorter timescale, they do have a mean towards which they tend to revert (it’s just that that mean does a random walk or other unbounded stochastic process on some much longer timescale).

    Thoughts?

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      It’s interesting when you go off the wall 🙂

      I guess on first principles I simply don’t believe that any real population is an unbounded random walk. (Of course bounded at zero, so let’s specify unbounded random walk on a log scale). On first principles there simply has to be density-dependence at some point (that’s not an empirically debatable point, it simply follows from finite Earth surface area if nothing else). And there is plenty of empirical evidence that even if you use a log scale there is absorption to zero from nearby for many species.

      So I’m not sure that it’s really interesting to pose that particular what-if. However, what if there’s substantial density vagueness so that most populations spend most of their time behaving as if an unbounded random walk? Hmmm…. Offhand, I think the regression-to-the-mean phenomenon still applies, as long as high level is more likely to be followed by a lower one than a still higher one.

      Like

      Reply
      1. Jeremy Fox

        Oh, I believe in density-dependence too! Guess I should’ve made that clearer. But I think my question could still be asked even in a word with density dependence. Imagine, say, a logistically-growing species in which K does a random walk that’s only bounded at zero.

        You’re probably right that the question as phrased is still too vague to be all that interesting. Might be more interesting to make it more concrete by looking at it in the context of some empirically plausible non-stationary population model. Wouldn’t be a hard thing to simulate.

        Like

        Reply
    2. Patrick Baker

      A somewhat related issue is that the quality of the habitat in which a species is found may change over time. We see this here in SE Australia with Leadbeater’s Possum, an arboreal marsupial once believed extinct. Long-term studies by David Lindenmayer have highlighted the decline, but the initial plots were established in areas of known occurrence and in forest considered high-quality habitat (lots of hollow-bearing trees for nesting and a dense Acacia understory for foraging and movement). However, the Acacia are short-lived (max age is ~70-80 yrs) and the forests, which established after a massive bushfire in 1939, are now 80 yrs old. The populations of possums on these plots are declining. However, it seems, contrary to the many publications on this species, that the cause of the decline remains an open question: is it a true decline, a decline in habitat quality leading to the possums moving elsewhere, a statistical artefact of setting up plots in areas of known high densities of a rare species, or some combination thereof…?

      Like

      Reply
  11. Jeremy Fox

    A further thought: your experience with this paper makes me pessimistic about the premise of the book I’m supposed to be writing. The premise of my book is that we can all learn how to do ecology better by taking a comparative approach. Look at lots of different cases of success and failure in ecological research programs and identify commonalities that pick out what works and what doesn’t. But your experience with this paper suggests that that comparative approach is doomed. I mean, it’s not as if regression to the mean is an obscure or tricky concept, or foreign to ecologists. Kelly and Price (2005) wrote a whole paper about it in frickin’ Am Nat, with both a general explication and several examples from various areas of ecology and evolution. And I’m sure the topic is covered in some/many undergrad stats courses. One would’ve hoped that these various discussions of the problem in other contexts would give many/most ecologists a “search image” that would allow them to recognize regression to the mean instantly whenever it crops up. But apparently not, if your experience with some of your reviewers is anything to go by.

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      I think that’s too pessimistic. I think we in part sampled by chance a larger-than-usual fraction of bad reviews; and in part we were talking about something emotionally charged (see Brian’s point). Yes, there are plenty of examples of the Boxer Effect in literature; but it doesn’t mean that it’s *everywhere*. I hope. So, write your book!

      Like

      Reply
      1. Jeremy Fox

        “I think that’s too pessimistic. ”

        You’re probably right. I tend to be too quick to assume that everybody is, or should be, very familiar with things with which I personally am very familiar. Relatedly, I get frustrated too easily at the speed with which some bit of knowledge diffuses to people to whom it’s relevant.

        Hmm, there might be a fun easy post here. Pose the question “What do you know that few others seem to know, even though it seems like they really ought to know and would benefit from knowing?”

        Like

        Reply

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.