I’ve devoted a lot of time and effort, over the last decade or so, to writing about good writing. There’s The Scientist’s Guide to Writing, of course; there’s our recent preprint on the construction of good titles; there are dozens of posts here on Scientist Sees Squirrel; and I can neither confirm nor deny rumours of another currently-super-top-secret book project. And this doesn’t even count the innumerable hours I spend toiling to improve my own writing, and to mentor my students towards improving their own.
Does any of this matter?
It seems obvious that it should, doesn’t it? Surely people would rather read well-written papers than cryptic or tedious ones? Surely, then, those well-written papers would have more impact on the progress of science (and the progress of their writers’ careers)? And of course I’m not the only one preaching, and working for, good writing – there are many other books, blogs, podcasts, and other resources all devoted to helping people write better.
Surprisingly, though, what literature there is on the matter is decidedly mixed. A number of folks have attempted to measure the quality of writing for a bunch of papers and associate it with their citation impacts. But these efforts haven’t given a clear story. Sometimes “better” writing is associated with higher impact, sometimes with lesser impact, and sometimes there’s no pattern at all.* The problem is, this literature is entirely observational. Yes, you can infer process from pattern; but you have to do so carefully. Good writing is likely to be confounded with many other things, so raw correlations of writing quality with citation record are probably not that helpful.**
Wouldn’t it be great, then, if we could do a randomized, controlled experiment in which we compare papers with better vs. worse writing that are otherwise identical? Yes, it would – and that’s what a new preprint from Jan Feld, Corinna Lines, and Libby Ross (an economist and two professional editors) does. It’s a clever study; I’ll summarize, but you should read the whole thing.
Feld et al. started with 30 papers written by economics PhD students. They then had professional editors revise the writing (but not the content) of each, so they had original and writing-improved versions. They checked to see that the edited version really was “better written” by having a panel of writing experts judge them (each judged 5 original and 5 edited papers, but none saw both versions of a single paper and none knew about the editing intervention).*** Sure enough, the edited ones were judged better written; that is, the experimental treatment “worked”.
But what did economics readers think? Feld et al. recruited a second panel, this time of disciplinary experts rather than writing experts, and asked them to judge the academic quality of the papers. Again, nobody saw both versions of one paper, and nobody knew about the editing intervention. And here’s the payoff: they rated the better-written papers as superior academically (admittedly, not a lot – about 0.4 points on an 11-point scale).**** Better writing helped sell the academic content of the papers.
Yes, good writing matters – and I can sigh with relief, having not wasted the time I’ve invested in trying to write better, and to help other scientists write better, too. You’ve probably made – you’re probably making – similar investments. If so, you can sigh right along with me.
© Stephen Heard April 12, 2022
Image: Not good writing. “It was a dark and stormy night…”, from Edward George Bulwer-Lytton’s novel Paul Clifford (1830). Check out similarly wretched prose at the Bulwer-Lytton Fiction Contest.
**^We faced exactly this problem in our recent study of humour in paper titles – we discovered that people give funnier titles to the papers that they themselves subsequently cite less – that is, their less important ones. As a result, the raw correlation of title humour with citation impact is negative; but after correcting for the confound, the actual effect of humour on citation impact is positive. You should read our preprint!
****^Is it a perfect study? Is any study perfect? I have a few minor quibbles. Both the writing-quality and academic-quality judgements were done quickly – in about 5 minutes per paper. I think that’s reasonable for writing quality; but it’s pretty superficial for academic quality. Feld et al. suggest that’s the kind of time someone might put in deciding whether or not to accept a paper for a conference or to decide on a desk reject at a journal. I hope this is wrong. And there was some heterogeneity in the kind of paper involves (micro- vs. macro-economics, empirical vs. theoretical) that fit a bit awkwardly into the way papers were grouped for scoring. But overall, this seems to me a strong paper, and I believe the results. Hopefully not just because I want to.