Photo: Two giraffes by Vera Kratochvil, released to public domain, via publicdomainpictures.net. Two giraffes are definitely better than one.
Ecologists are perennially angst-ridden about sample size. A lot of our work is logistically difficult, involves observations on large spatial or temporal scales, or involves rare species or unique geographic features. And yet we know that replication is important, and we bend over backwards to achieve it.
Sometimes, I think, too far backward, and this can result in wasted effort.
Which brings me to the oddness of the number two. I’m sure you’ve read this study, if you haven’t written it yourself: some experiment or some observation is conducted, and then repeated in a second year. Or perhaps the experiment is replicated at two different study sites, or in two plots at a site, or for two different species. (I’ve done this myself; as just one example, this study is replicated at two different sites, although only in a single year.) And yet: two seems, in general, like a really dumb choice.
If I run an experiment in two different years (I’ll stick to “years”, but the argument is identical for sites, species, etc.), I might be thinking about years in one of two different ways*. First, I might be aware of some interesting difference between the two years – for instance, one was a drought year and the other a “normal” year. Second, I might merely be aware that the world is a variable place, and have some expectation that the outcome of the experiment might vary between any two years. These ideas about year-to-year differences are treated differently, statistically – but in both cases, it seems like two is a dumb number.
In the first case, “year” is what’s known as a fixed effect. I’m explicitly interested in the contrast between the drought year and the normal year, and I’d like to estimate the effect of drought on whatever we’re measuring. But of course my two years give me one drought year and one normal year. With enough replication within years, I can get a really good estimate of the difference between those years – but I can’t ascribe that difference to drought (or anything else) because there’s no replication at the level of drought vs. normal. So two is really no better than one – despite being twice as much work and expense.
In the second case, “year” is what’s known as a random effect. I’m conceptualizing different years as essentially random draws from a larger universe of years that I might have studied, with the idea that there’s variation among years in the outcome of my experiment. This time, I want to estimate the among-year variance component (wanting to know, perhaps, whether it’s big enough that no single-year study means much). But estimating variances is hard, and estimates of variance based on two data points are very close to meaningless. So again, two is really no better than one.
And yet, sample sizes of two (years, plots, sites, species, etc.) are abundant in our literature**. Why? Not, I think, because we’re collectively unaware of the futility of two. I doubt that anything I’ve written so far has surprised you. My best guess is that despite two’s statistical futility, it nonetheless plays a useful role in an effective publishing strategy. We all know that reviewers will question “unreplicated” studies, and so we repeat an experiment in two years simply to pre-empt that criticism. Reviewers and editors tend to be satisfied by two years, not because they really think it improves our inference, but because it (1) brings a study into alignment with common literature practice, and (2) gives them enough cover that they don’t have to think of themselves as approving an “unreplicated” study. We’re all parties to this unspoken social contract: we agree to pretend that two is enough, and we agree to ignore all the other axes along which the study might be “unreplicated”. (Did we repeat it with two genotypes? In two different months in each year? With two different brands of fertilizer? Using two different capture or marking techniques? I could go on; but you’d probably prefer I didn’t.) We implicitly agree on two because it smooths our publishing interactions***.
Is this cynical? Yeah. Am I worried that my next manuscript may be rejected by a reviewer who points out we’ve only done two years, and cites this post? Absolutely.
© Stephen Heard (firstname.lastname@example.org) July 25, 2017
*^If I’m “thinking” about years at all, but I guess part of my point is that I’m probably not, really.
**^I was starting to sift through my own papers to see how many times I’ve done this myself. But before I got far, I realized that I’d rather not know. I know, that makes me a bad person.
***^If you watch The Big Bang Theory, you’ll know that Sheldon can object to some action because it’s objectively and logically silly (gift-giving, for instance). But if he’s told that it’s a “non-optional social convention”, he simply shrugs and climbs happily on board. Two may simply be a non-optional social convention in scientific publishing.