The efficiency of the lazy scientist

Photo: Lazy red panda CC 0 via pxhere.com

I’ve just published a paper that had some trouble getting through peer review.  Nothing terribly unusual about that, of course, and the paper is better for its birthing pains.  But one reviewer comment (made independently, actually, by several different reviewers) really bugged me.  It revealed some fuzzy thinking that’s all too common amongst ecologists, having to do with the value of quick-and-dirty methods.  Quick-and-dirty methods deserve more respect.  I’ll explain using my particular paper as an example, first, and then provide a general analysis.

For our paper*, we needed to estimate plant fitness (or more precisely, how fitness was reduced in plants under insect attack).  But we were working with goldenrods, which are perennials that reproduce both asexually (through underground rhizomes) and sexually (through outcrossed seed).  So what’s the best measure of fitness?  That’s a question on which reviewers had (and shared) some strong opinions.

Here’s how we estimated plant fitness: we did something very quick-and-dirty.  We simply clipped plants at ground level and measured the total dry mass of their aboveground tissue**.  Since only aboveground tissue produces the photosynthates (sugars) needed for construction of all other tissues, aboveground biomass should correlate (perhaps imprecisely) with pretty much any other fitness measure one might choose.  (Or so we argued; and there are data for goldenrods to back us up.)   But some reviewers were horrified, and although none of them actually used the word “lazy”, it was pretty obviously there between the lines.

Instead, the reviewers argued, we should have dug up and counted rhizomes, and harvested and counted seeds.  There’s no argument that those would be useful measures of plant fitness (via asexual and sexual reproduction, respectively) – even, I’ll grant you, much more precise measures than aboveground biomass.

The thing is, counting rhizomes and seeds is hard. I know; I’ve done both.  Digging up rhizomes isn’t awful in a greenhouse experiment, but a mature old-field turf (where we worked) it’s extraordinarily laborious at best.  Counting seeds is horribly time-consuming, because goldenrods make thousands of tiny wind-dispersed seeds distributed among hundreds of bract-enclosed flower heads.  We could have counted rhizomes and seeds anyway, of course – but here’s why we didn’t.  We had a limited budget for our work (both in time and money).  Everyone does.  In the time it would have taken to count rhizomes and seeds for one plant, we could measure aboveground biomass for 100 (at least).  That’s the thing about quick-and-dirty: it’s quick.  And that means there’s an important question: are we better off with one precise measurement, or 100 imprecise ones?

Here’s where the fuzzy thinking comes in: our reviewers didn’t talk about the choice between one precise measurement and 100 imprecise ones.  Instead, they just complained that our measure was imprecise, with the implicit but false premise that we could have substituted a precise measure one-for-one.  But it’s important to frame the 1-vs-100 part of the question explicitly, and having done that, we can answer it.  So now, a more general analysis.

Imagine that you want to estimate the value of T, some biological quantity with an unknown but presumably fixed and finite value.  (You can think of T as standing for “True”.)  We can’t measure T directly.  However, we can take multiple measurements of either H, which is hard to measure but contains quite a bit of information about T, or E, which is easy to measure but is much vaguer about T.  What should we do?  Reviewer-pleasing H, or quick-and-dirty E?

I’ve set up simulations to explore this question.  I’ll show just one numerical example, but here’s the R code, so you can play around to your heart’s content.  Here we go:

• The true value we’re after is T = 100.
• Hard-to-measure H is a random normal variate with mean 100 and variance 4 (that is, each measurement of H is a relatively precise estimate of T). Easy-to-measure E is another random normal variate, this time with mean 100 and variance 64 (each measurement of E is a relatively imprecise estimate of T).
• Each replicate of an experiment to estimate H costs $100, while each replicate of an experiment to estimate H costs $5. (Note that nothing changes if our currency is time instead.)
• We have a total budget of $1000, so we can measure H 10 times or E 200 times. In either case, we take the mean across measurements, which is our best estimate of T.
• I simulate each strategy 10,000 times (that is, 10,000 estimates of T each based on 10 measurements of H; and 10,000 estimates of T each based on 200 measurements of E).

Here’s how that comes out:

The top plots show the distributions of values of H and E.  Both are centred on the true value T, of course; but H values are bunched tightly around T, while quick-and-dirty E estimates are broadly spread.  Nothing surprising there.

The bottom plots show the distribution of T estimates if we spend our budget on a few very precise measurements (via H, mean of 10 measurements) or a lot of quick-and-dirty ones (via E, mean of 200 measurements).  In my example, the quick-and-dirty method is better.  (See, reviewers?  See?  Sigh.)  In this case, the quick-and-dirty measurements had 16 times the variance of the precise ones – but they were 20 times cheaper, and the replication that allowed gave them the edge in estimation.

Did I set up the numbers to make the comparison come out the way I wanted it to?  Well, sure; but it wasn’t that hard.   It all depends on the ratio between the cost difference and the variance difference between the two measures H and E***.   What may not be intuitive is that for perfectly realistic (in ecology) cost ratios, the properties of statistical estimation are such that quick-and-dirty measures will outperform precise ones.  In my goldenrods, it’s at least 100 times more costly to count rhizomes and seed than to measure aboveground biomass (with 100 being a conservative estimate).  As a result, I could crank up E’s variance to 400 (in my example) and still come out at least even****.  In other words, if quick is quick enough, dirty can be really dirty.

So, if you’re ever tempted to cast aspersions on a quick-and-dirty technique, please do what my reviewers didn’t: think twice.  Sometimes, an apparently lazy scientist is actually just happy to let mathematics do the work for free (via E and the magic of statistics).  Sometimes, a lazy scientist is just an efficient one.

© Stephen Heard  January 29, 2018

---

*^Our paper, which was the MSc thesis of my excellent student Yana Shibel, has to do with the evolution of herbivore impact in novel plant-insect interactions.  We tested the hypothesis that after a new plant-herbivore interaction is established, selection will favour reduced impact of herbivore on host plant. If you’re curious about that, you can find the paper here; and if the paywall is a problem for you, just email me and I’ll send a copy.

**^This is not an uncommon technique, and often works fairly well, according to a providentially-recently-published review.

***^You can solve all this analytically, which I leave (as we love to say) as an exercise for the reader.  It comes out quite simply.  Given normality assumptions, if you double the variance, you need to double the number of estimates to preserve estimation performance.

****^Or actually, way ahead, given the sanity-destroying nature of excavating rhizomes and counting seeds.

Advertisements