Image: Excerpt from Heard et al. 1999, Mechanical abrasion and organic matter processing in an Iowa stream. Hydrobiologia 400:179-186.
Nearly every paper I’ve ever written includes a sentence something like this: “All statistical analyses were conducted in SAS version 8.02 (SAS Institute Inc., Cary, NC)”* But I’m not quite sure why.
Why might any procedural detail get mentioned in the Methods? There are several answers to that, with the most common being:
- because it allows someone else to replicate our experiment;
- because it establishes authority of work that’s the credible product of an authentic scientist; or
- because it helps the reader understand the Results.
(There’s a 400-year history of hair-rending and teeth-gnashing over which of these matters most, and there’s a good case that we don’t act consistently with our beliefs about this. I’ve explored the issue in an older blog post and in more depth in The Scientist’s Guide to Writing.)
Let’s apply these criteria. There’s a good reason you’ve never seen this sentence in a paper:
Data were recorded with a PentelTM #2 0.5 mm mechanical pencil on quad-ruled 8.5 × 11” Rite-in-the-RainTM paper.
Knowing this doesn’t meaningfully help someone replicate your experiment (although drawing a clear line under the list of details that are needed to “replicate” points out how peculiar our discourse can get around replication). It won’t persuade anyone that our work should have authority, and it won’t help the reader understand the Results. So we keep these details to ourselves**.
How does our stats-software sentence fit with these possibilities? It certainly doesn’t help the reader understand the Results: a 2-way ANOVA is a 2-way ANOVA, whether it’s executed in R, SAS, SPSS, or (gasp) Excel. There are probably readers for which it can help establish authority – but there shouldn’t be. These readers are the people who believe that if you aren’t using R (for example), you’re not an authentic scientist. I hope we can all agree that nobody should believe this***. Finally, what about replication? This point is a little more nuanced, so let’s think about it a bit.
If I ran some “standard stats”, like a 2-way ANOVA, a principal components analysis, or a logistic regression, then you don’t need to know which software package I used in order to replicate my work. Standard stats are like pencils and paper: I use them, but you don’t care that I did; if you want to replicate what I did, you’ll use your own and it won’t make any difference. “Exotic stats” are different. If I ran “exotic stats”, perhaps I invented a new test, or used a method recently published for which there remains some doubt about its performance or even correctness – BiSSE-class models, for example. Here someone really might get different answers using one R package versus another (for example); and so we really do need to report the stats software we used. (By the way, one mark of good experimental design is that it puts the weight as much as possible on the simplest, and thus most standard, stats.) Where’s the line between standard stats and exotic stats? Ah, that’s a bit tricky – but we make judgements like this all the time with respect to other methods. Does it matter what our vials were made of? No, if we’re storing insect specimens for morphological measurement; but yes, if we’re storing them for analysis of cuticular hydrocarbons. We’re smart; authors can decide, reviewers can question, and we can get this right.
But that isn’t what we do. Instead, in ecology and evolution we seem to have a de facto standard that we report our stats software, even if we used nothing fancier than one-way ANOVA. That’s weird. Just as weird, in cell biology, the standard seems to be not to mention stats software. Neither makes any sense.
So am I going to put my money where my mouth is, and stop bothering my reader with trivia like whether I calculated correlation coefficients in R or in Excel? No, because there’s another completely different reason one might mention a stats package, and it’s one I can get behind. Mentioning a stats package gives me an opportunity to cite it. Citation has a number of functions, but the one I care about here is as a currency of appreciation. When I cite a stats package, I thank its authors and give them a tiny but tangible reward (a CV boost). This doesn’t matter to me for commercial packages, so if I do stats in SAS I don’t feel the need to cite for appreciation. It does matter to me for software written by one of my colleagues as part of their contribution to science, so when I do stats in R I cite both base R and any add-on packages. This practice may be bizarrely inconsistent seen through the lens of the function of the Methods, but it’s entirely logical seen through the lens of science as a community. Getting the right lens in place makes all the difference.
© Stephen Heard (firstname.lastname@example.org) November 7, 2016
This post is based, in part, on material from The Scientist’s Guide to Writing, my guidebook for scientific writers. You can learn more about it here.
*^I’m trolling you a little. These days most of my papers specify “R version something-or-other” instead. Trust me, really, I belong to the 21st century – even if it does cost me the occasional bottle of wine supplied to younger, R-hipper members of the lab. Of course, my results are unaffected.
**^Although I’ve reviewed manuscripts specifying the brand of calculator used to make calculations, the brand of -80º freezer used to preserve specimens, and the size of vial used to hold specimens preserved in ethanol. I was as mystified as you surely are about each of these.
***^Cults are cults (speaking of trolling), in science as in everything else. I once had a reviewer argue that my work wasn’t publishable because I did analyses using code I wrote in Microsoft Visual BasicTM – on the grounds that even though I provided the code, the compiler isn’t open-source. Sigh. This is not, of course, to say there’s anything wrong with R as a statistical tool; only that there’s nothing uniquely right with it, either.