Reproducibility, your Methods section, and 400 years of angst

(Image: Robert Boyle’s (1660) vacuum pump, from New Experiments Physico-Mechanical, Touching The Spring of the Air, and its Effects; Made, for the most part, in a New Pneumatical Engine)

Unless you’ve been living under quite a large rock, you’ve heard or read a lot lately about the “reproducibility crisis” in science (here’s a good summary). That our work should be reproducible is certainly a Good Thing in principle, but there are complications where the rubber hits the road. Today, some thoughts on reproducibility, and on what, if anything, it means for the writing of a paper’s Methods section.  And I think some historical perspective is both interesting and useful – because the reproducibility “crisis” is 400 years old.

There’s an odd disconnect in the way we think about our Methods sections.

Advertisement

Plant galls: how insects co-opt plant development to build themselves homes

(Image credit: Lahvak via Flicker/CC BY-NC-SA)

Note: This is a science outreach piece belonging to a series I wrote for the newsletter of the Fredericton Botanic Garden. I’d be happy to see it adapted for use elsewhere and so am posting the text here under a CC BY-SA 4.0 license. If you use it, though, I’d appreciate hearing where and how.

---

With winter upon us, a walk in the Garden has gotten less colourful: nothing is in flower, and most plants have died back or dropped leaves. But I hope you’ll see this as an opportunity to notice things that aren’t as easy to spot in the full flush of summer vegetation. Plant galls are such a thing, and they’re a piece of natural history that especially fascinates me. A “gall” is an abnormal growth on a plant, caused by an attacking natural enemy such as a bacterium, a virus, or an insect herbivore. I’m a particular fan of insect galls, because they tip us off to a complex web of developmental, ecological, and evolutionary interactions between the insect and its plant host.

What’s your most overcited paper?

“Publish or perish”, we say, except that it probably isn’t enough just to be published: we want to be, and maybe need to be, highly cited.  Tenure committees, granting agencies, and the like devour citation data, journals compete for citations to boost their impact factors, and we worry about detecting authors who self-cite to manipulate their citation stats.  Now, all this may sound like a lead-in to a post decrying overemphasis on citation counting, but it isn’t.  Actually, I think citation counting is worthwhile – so long as it isn’t fetishized*.  After all, a paper with lots of citations probably made some people think, and with luck had some influence on the progress of science (a nice post on this from Pat Thomson is here).

Our emphasis on citation means that we are (I think) all very aware of the citation performance of our own papers.  It’s easy to track via Web of Science or Google Scholar, and that’s how I made the figure above: citations vs. years post-publication for 65 of my own papers, taken from my Google Scholar profile.  There’s a lot I could do with these data, but for some reason I’ve been thinking about which of my papers is the most overcited.  (I hope it’s clear from the title that I want you to mention your own most overcited paper in the Comments.)

What could I mean by an “overcited” paper? 

In defence of the P-value

(graphic by Chen-Pan Liao via wikimedia.org)

The P-value (and by extension, the entire enterprise of hypothesis-testing in statistics) has been under assault lately. John Ioannadis’ famous “Why most published research findings are false” paper didn’t start the fire, but it threw quite a bit of gasoline on it. David Colquhoun’s recent “An investigation of the false discovery rate and the misinterpretation of P-values” raised the stakes by opening with a widely quoted and dramatic (but also dramatically silly) proclamation that “If you use P=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time.”* While I could go on citing examples of the pushback against P, it’s inconceivable that you’ve missed all this, and it’s well summarized by a recent commentary in Nature News. Even the webcomic xkcd has piled on.

Wonderful Latin Names, Part II: Abudefduf saxatilis

(Image by Klaus Rudloff, kdrudloff@web.de, via http://www.biolib.cz)

Today’s Wonderful Latin Name is that of the Sergeant Major: Abudefduf saxatilis. The Sergeant Major is a common damselfish of the tropical Atlantic, often the first fish a new snorkeler or diver learns because it’s beautiful, abundant, and distinctive. Like the hoopoe (Upupa epops, the subject of Wonderful Latin Names, Part I), the Sergeant Major has a Latin name that’s both fun to say and etymologically interesting. I was first drawn to the name because of its peculiar rhythm: Abudefduf saxatilis. It seems to have a time-signature change between genus and species, which arises because the species name is conventionally Latin, while the genus name is Arabic.