Arithmetic, intuition, and very large errors

Here in Canada we’ve just had a federal election. As politics often does, it put on display two kinds of people: those whose thinking has led them to have strong opinions, and those whose strong opinions have led them to stop thinking. I saw a stunningly good example of the latter group, and the amusing story carries a message that applies much more broadly. So here goes.

Our election was called by our Prime Minister (I’m skating by a bunch of technicalities here) rather than occurring on a schedule or following the fall of a government. It was hard to see any non-flagrantly-political reason for the election call. Elections, of course, cost money to run, and at one point a news article breathlessly reported a cost estimate of $600,000,000 (six hundred million dollars, in your best Dr. Evil voice). Those who didn’t want an election seized on this number, and one person in particular* broadcast on Twitter: “Let that sink in: every INDIVIDUAL in Canada could get $10 million worth of programs and services, and there’d be loads of money left over”.

Shocking, right? But clearly, obviously, wrong. The actual number is about $16. That’s right: not $10 million, but $16. That’s an error of just under six orders of magnitude, and the $10 million claim is so utterly implausible that you wonder how anyone could have believed it for more than a split second.

I can think of two possibilities.

One is that the person who made the error had no mathematical common sense at all. It wouldn’t take more than a tiny bit of intuition: it just isn’t plausible that ANYTHING a country does (or even EVERYTHING a country does in a whole year, for that matter) can cost every citizen $10 million. After all, the vast majority of citizens don’t have – and will never have – $10 million to (hypothetically) spend!** But if that doesn’t immediately get you, how about coming at it the other direction: if the cost figure is $600 million, then it only takes 60 people paying $10 million each to pay the whole freight. Or, perhaps (like me) you don’t actually know the current population of your own country, but are pretty sure it’s between 30 and 40 million – so a quick 600/30 means the answer can’t be more than $20.  So there are at least 3 different ways for intuition, or a bit of arithmetic so simple it’s almost intuitive, to save you. Now, my very weird brain does this kind of thing without my needing to think much about it, and not everyone’s brain is weird in the same way mine is. But this kind of thinking is something you can deliberately practice – and, if you’re given to making quantitative pronouncements, is something you really should practice.

The other possibility is both more interesting and more important. It’s that the person who made the error had mathematical common sense, but had it switched off. What do I mean by that? Just that the $10 million mistake was a very convenient one: it confirmed their prior belief that the election call was not just bad, but outrageously bad. I think a lot of us prefer not to question information that confirms our prior beliefs. I know, that’s not a stunningly original observation; but I’m not sure I’ve ever seen such a spectacular example! And scientists have this bad habit too.  We often think of science as a system for learning things that’s explicitly designed to separate data from prior belief – and indeed, this is part of the reason we have inferential statistics (among other things). And yet… who hasn’t seen Reviewer #2 object to a study because the results are surprising? Who hasn’t heard someone say “Hmmm, that’s odd, better run that experiment again”?

The moral of the $10 million story, I think, is a simple one: if a number, or a piece of evidence, is surprising, check it; but check it even if – no, check it especially if – it’s surprising in a direction that confirms something you already think. In other words, be skeptical of any new result that makes your prior beliefs seem very clever. Because it’s often when we feel cleverest that we’re actually being the most dense.

© Stephen Heard  September 28, 2021

Image: Big Mistake © Double:Zanzo via flickr.com CC BY-NC 2.0


*^No, I’m not going to identify them. As you’ll see reading on, they made a very large mistake; but it’s the kind of mistake that gets made a lot. Asking which particular person made it so we can all shame that individual misses the point in a rather spectacular way.

**^Skating past some technicalities of means vs. medians and the inequality of wealth distribution here; but an attempt to get closer than six orders of magnitude to the real answer isn’t going to founder on those technicalities.

 

 

2 thoughts on “Arithmetic, intuition, and very large errors

  1. Pavel Dodonov

    “if a number, or a piece of evidence, is surprising, check it; but check it even if – no, check it especially if – it’s surprising in a direction that confirms something you already think.” – this is my approach to statistical analysis. If a p-value is very, very low, I immediately think that something is wrong with the analysis. And quite often I’m right. 🙂

    Like

    Reply
  2. egd11

    I’ve had a few disturbing experiences with doctors leaping to clinical conclusions based on an wrong reading from an instrument (obviously wrong if you stop to think for a minute) – so I’d add the importance of doing basic common-sense checks of numbers that come from instruments that could malfunction or be misread. No only in research but also in applied contexts.

    Like

    Reply

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.