I made scones this morning, and it made me think about statistics, and about thinking. No, really, I have a point: it’s that P = 0.05 and a teaspoon of baking powder are the same thing, in an important way. Am I stretching an analogy to its breaking point? Read on to find out.
My scone recipe calls for 4 cups of flour, a cup of sugar, a teaspoon of baking powder, a teaspoon of baking soda, half a teaspoon of salt, four tablespoons of butter or shortening, and then raisins and buttermilk to make a dough.* The quantities are interesting. People will tell you that baking needs precision, with carefully measured ingredients. Can that really be true? Notice the nice round quantities: 1 teaspoon of baking powder, not 0.75 or 1.2 or 1.37928. And a quick flip through my recipe file suggests that nearly every recipe uses similarly nice round quantities.** If precise measurement really was important, at least some of the time the required amount of some ingredient would refuse to be a nice round multiple of 1 teaspoon. Conclusion: people measure baking ingredients carefully because they’ve been taught to do so without thinking about it, not because effective baking requires precision.
What about P = 0.05? Same thing, really. We teach undergraduate statistics poorly, and our students (in my experience) emerge thinking that P = 0.05 is a really, really important threshold. Actually, it’s not just our students – think about the ubiquity of jokes and snide comments that come up whenever anyone has the temerity to discuss a pattern that, when tested, yielded P = 0.06. But of course, there’s nothing fundamental about P = 0.05. It’s just an arbitrary stab at a compromise between rates of Type I error (which increase as you use a less stringent criterion) and of Type II error (which increase as you use a more stringent one). It’s perfectly fine to use an alternative threshold if you have reason to prefer a different resolution to the Type I/II tradeoff, or to use no threshold at all and interpret P as a continuous measure of evidence against the null hypothesis. And if you think carefully, those things should be obvious from the fact that we use such a round number (0.05, or 1/20). If there were a fundamentally meaningful number for a significance criterion, one emerging from the fabric of mathematics rather than from convenience for our own practice, it would surely be something more like 0.04823692. (There’s a reason that neither π nor e = 3). But thinking carefully about statistics is, sadly, something we generally don’t train our undergraduates to do.***
To be clear, none of this makes the P value useless or unimportant. It just makes it obvious that we don’t often think very clearly about it. As scientists, we tell each other that we value clear and critical thinking. I’ve pointed out before that there seem to be many ways in which don’t walk that walk. Reverence for P = 0.05 is one, and you can get to that conclusion via thinking about my scone recipe. See? There really was a connection.
So: if you’re willing to do a little thinking, baking needn’t be precise; you can skimp on the baking powder or on the sugar, and see what happens. The baking gods won’t smite you for your temerity. Same for P = 0.05. The statistics gods won’t hurl thunderbolts if you think carefully about why a result with P = 0.052 is noteworthy, or a different result with P = 0.048 isn’t. Mind you, in lieu of those thunderbolts, you should probably prepare yourself for some strongly-opinionated-but-poorly-thought-out mockery. Cuing my Twitter feed in 3, 2, 1….
© Stephen Heard January 4, 2022
Image: Ludicrous precision. Own work, CC BY 4.0.
*^Since I just gave you most of the recipe, I might as well finish it. I like about ¾ c raisins, and it will take about 1¾ c buttermilk to make the dough workable. Pat out on a counter and dust tops with another half tablespoon or so of sugar; cut into about 10 scones and bake 25-30 min at 350˚F on a greased cookie sheet.
**^With a couple of exceptions. First, pie crusts seem to use peculiar quantities, like 7 tablespoons of butter – suggesting that precision actually is important for pie crust. (In my experience, so is skill. I do not make pie crusts.) Second, yeasted baked goods almost always call for 2¼ teaspoons of yeast. But that ¼ doesn’t matter – its roots are in packaging, not performance, and a while ago they let me make another strained analogy between cooking and science. I am nothing if not a shameless recycler of ideas.