Barf-and-buff writing, and cherry-picking citations

I’m a big fan of a writing strategy that, in The Scientist’s Guide to Writing, I call “storming the beach” – but that’s sometimes more vividly termed “barf and buff”.  The idea is simple: early in a writing project, don’t stop to make things perfect. Instead, charge ahead with getting something – anything – on the page. Rough, awkward, incomplete – it doesn’t matter, you can fix it later. Did you write some crap? That’s OK: you can fix crap much more easily than you can fix a blank page. So barf out something terrible, and buff it later.

Like most good advice, “barf and buff” has a few dangers lurking in it. One of them has become A Reason I See Red On Twitter. (Yes, there are quite a few of those.)  Here it is: barf-and-buff is not an invitation to cherry-pick citations.

A key piece of barf-and-buffing is that when you’re storming your way through a draft and there’s something you need to look up, you don’t stop. Want a better word for “obvious”? Can’t remember the reaction temperature? Need the lat/long for a study site? Unsure when Herschel discovered Uranus?* Don’t stop. Just flag the spot for later attention (I use three asterisks, which is easy to spot or search for later) and keep going. You can do your looking up when you’ve come to the end of your first draft and momentum isn’t so precious.

One of the things I often flag for later attention is a citation, but here’s where the danger lurks. It’s one thing to know you’ve read a paper showing that thing X is true, but you don’t have it at your fingertips right now. If so, sure, write “Cats are smarter than dogs (***Find that reference***)” and move on.  It’s quite another thing to decide to make a claim now, and locate support for it later (“Cats are smarter than dogs (***Find a reference)”.

This is where my seeing red on Twitter comes in. Far too often – every day, it seems like – I see someone posting a question like this: “Can anyone point me to a citation for cats being smarter than dogs?”  I’m not sure you could signal more clearly that you’ve already made up your mind, and are content to cherry-pick the literature to find support for the position you’ve already decided to hold. Of course, the question one should ask is “Can anyone point me to some citations on the relative intelligence of cats vs. dogs” – but tweets like that are, sadly, uncommon.

You might, of course, protest that if it turns out you’re wrong, and cats aren’t smarter than dogs,** you can simply change what you’ve written. That’s completely compatible with the barf-and-buff strategy, after all. But three things. First, asking for help with cherry-picking makes me doubt your openness to doing that. Second, the nature of doing science on the very complex system that is our universe means that if you dig hard enough, you can almost always find a citation to support a given statement. And third, it’s pretty common human psychology to be a little bit attached to the first way you wrote something, and a little bit resistant to making a major overhaul. You can of course (and I have), but writing in Claim X makes the path to writing it out again distinctly uphill.

So, please do barf and buff: it’s an excellent strategy for productive writing. But don’t extend it to shortcutting your literature review. If you must, write “Cats are smarter than dogs (***are they? Do a literature search***)”, to keep yourself honest. And if you crowdsource your literature search, please don’t ask people to become accomplices in your cherrypicking.

And now, if you’ll excuse me, I have some literature searching to do.

© Stephen Heard  February 2, 2021

Image: This kind of cherry picking is OK. © Matt Ryall via Wikimedia.org CC BY 2.0


*^March 13, 1871. But if you came to this footnote expecting a rude little joke about that planet’s name, well, you’ll have to supply it yourself.  Although I have a frog you might be interested in.

**^I know, my example is a bit silly. There’s no possible universe in which cats aren’t smarter than dogs. Hang on, I’ll find a citation for you soon.

8 thoughts on “Barf-and-buff writing, and cherry-picking citations

  1. Dr. Catherine Scott (@Cataranea)

    I would argue that this issue is not quite as black and white as “find that reference” vs. “find *a* reference.” Here’s an example of a situation where you might say I had already made up my mind that spiders are important prey for birds and tweeted a question to help find literature: https://twitter.com/Cataranea/status/1231593050578849799?s=20, indeed, I wrote, “I’m pretty sure that… but I don’t have references at my fingertips.” But if my search failed to bring up any literature, or very few papers, then I would be forced to adjust my position.

    Also, when I am writing a first draft and trying to “barf” and not stop to look things up, I will often make a statement like the following actual one from an old draft that I just found: “Males lose body condition during mate searching (IS THIS TRUE?).” As I am writing, I am not necessarily thinking of a particular citation that I recall showing this, but making a general statement about. something that I think is likely to be true based on my broad understanding of a subject. Then when I’m “buffing” I might go on twitter and ask “does anyone know of any papers showing that males lose body condition during mate searching?” You would presumably interpret this as me looking to cherry pick citations to confirm this statement, when in fact I am not at all sure ahead of time that this statement is in fact true (or that there is much evidence to support it).

    Liked by 1 person

    Reply
    1. ScientistSeesSquirrel Post author

      I also use the “IS THIS TRUE?” marker when I’m writing!

      I don’t think we’re far apart. As you say, one key is that you recognize you may have to adjust your position. For your second example, though, why not as “does anyone know of papers showing WHETHER males lose body condition…”? Even if you have no intention of cherry-picking, I think the “showing THAT” phrasing invites others do do it on your behalf, doesn’t it?

      Thanks for commenting!

      Like

      Reply
      1. Dr. Catherine Scott (@Cataranea)

        I think asking for papers “showing that X is true” vs. “whether X is true” has a lot to do with norms of how we formulate hypotheses and “sell” papers and the bias in what gets published (we rarely see papers titled “no evidence for an effect of X on Y.” I remember as a grad student being corrected repeatedly when I would formulate my research objectives as questions, e.g. “does mate searching affect male body condition?” My committee would say, no, you need to formulate this as a hypothesis with specific predictions (“I hypothesize that mate searching affects male body condition; I predict that males will lose condition during mate search”).

        Also if someone on twitter asks “do you know of any papers showing X” and I know of papers that show Y, or fail to show X, I would respond by sharing those papers, because I would give the person asking the benefit of the doubt that they are looking for information, not intending to cherry-pick.

        Liked by 2 people

        Reply
        1. ScientistSeesSquirrel Post author

          You might well be right about that connection. Hypotheses with predictions are indeed important, but they support the asking of questions rather than replacing that! I think sometimes we get a bit carried away with hypotheses (I can hear people gasping as they read that…)

          Like

          Reply
  2. Jeff Houlahan

    Hi Steve, this is an example where I think the problems around ‘cherry-picking’ are overstated. For starters, nothing you say in your intro will change what your data show – so whether there was a great foundation motivating your study really doesn’t matter because the lasting value of your paper will be in the data. Second, there are no rules about where hypotheses should arise – whether it’s because one paper out of a hundred suggested something you want to investigate or 99 out of a hundred, really doesn’t matter – the strength of your inferences rely on your data not on anything you wrote in the preamble. Third, an introduction is rarely a meta-analysis and the synthesis is therefore always an idiosyncratic mix of what and how much literature you decide to read and how you decide to weight the literature – one person’s ‘cherry-picking’ is another person’s ‘considered interpretation’. Lastly, often we’re making a minor point and time spent ensuring your citations are somehow ‘representative’ of the literature is time that could arguably be best spent elsewhere. So, what actual damage to science arises from ‘cherry-picking’?
    Look, I agree that there is room for debate about ‘best practices’ in science as relates to typos, using passages plagiarised from your own methods sections and ‘cherry picking’ citations (yes, I’ve gone back and plucked a couple of other places where I think you and I disagree) but the idea that the line separating scientists on these issues demarcates anything substantive is one I would have to see evidence for.
    Not that you’ve said that – so, to some extent (and maybe a large extent), I’m putting words in your mouth. If these are pet peeves – fair enough. But, if this is intended to give advice on ‘best practices’ – I’m on the dissenting side.
    The argument tends to be – “hey if you’re sloppy anywhere, you’re probably sloppy everywhere”. I’ve never seen any empirical evidence of this.

    Jeff

    Like

    Reply
  3. Dr. Catherine Scott (@Cataranea)

    I was just thinking about this again, and would like to make another argument for why someone going on twitter and asking if anyone knows of citations “showing X” maybe shouldn’t necessarily make you see red. I actually think that asking that question on twitter may be *less* likely to result in cherry-picked citations than not asking at all. Using my “I’m pretty sure birds are important predators of spiders” example, if I just set out to search for citations on my own, I might search for “birds + prey + diet + spiders” or something similar in google scholar (because I am already pretty sure I’m right, and that will bias my search), and perhaps find a paper supporting the idea that birds prey on spiders. But if I ask my question on twitter (assuming my circle of friends and followers is sufficiently broad), some know-it-all is likely to see it and chime in with, “well actually, here’s some literature showing that rodents are the most important predators of spiders”, or “here’s a review of the bird diet literature showing that spiders are relatively unimportant.” Similarly, if I search for literature about how mate searching affects male body condition in Scholar, I’m likely to find papers that found this to be true (and the studies that failed to do so are much more likely to never have been written up and submitted), whereas someone on twitter might respond with “hey, I study mate searching and based on my knowledge of the literature, effects on body condition are pretty rare.” At least, my experience is that people love to “well actually…” and explain why other people are wrong on twitter, rather than just answer the question they asked.

    Liked by 1 person

    Reply
    1. Jeff Houlahan

      Catherine, I agree with much of what you say – but I think my point is a little different than yours. Even if we could distinguish cherry picking from a more considered approach to finding a citation – what harm is caused by cherry-picking? And maybe more importantly – what’s the benefit of doing otherwise? We know the cost.

      Jeff

      Like

      Reply
  4. Pingback: What jigsaw puzzles could have taught me about writing – if I’d listened | Scientist Sees Squirrel

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.