Why I sign (most of) my reviews

A few months ago, I wrote a post that prompted a brief twitter discussion with Meghan Duffy about whether we sign our reviews. I tend to sign mine, and Meg tends not to, but neither of us felt completely sure that our approach was the right one. So, we decided that it would be fun to write parallel posts about our views on signing (or not signing) reviews. Here is Meg’s, over at Dynamic Ecology; please read it, as she makes excellent points (all of which I agree with) even while arriving at a different conclusion (and a different default practice) than I do!

A lot has been written about the merits of signed vs. anonymous peer review.  There are arguments on both sides (which I don’t intend to review comprehensively), but in general I’m firmly convinced that at least the offer of anonymity is important to getting broad reviewer participation and high-quality reviews.  But I sign almost all of the reviews I write.  This seems odd in at least two ways.  First, here I am plugging anonymity, but I don’t use it much; and second, if I sign almost all of my reviews, why don’t I sign all of them?  I’ll try to explain; and I’m trying to explain to myself as much as I am to you, because I’m far from convinced that I’m doing the right thing.

My reason for signing is not a belief that the review process should be open and transparent in general – that a culture of signing reviews make them “better”, either more constructive or less likely to contain ad hominem attacks or other reviewer misbehaviour.  You hear a lot of arguments to that effect, but I’m not convinced by them*.  I’m unaware of strong data suggesting that signing improves reviews, or if does, that it’s the only way to achieve that.  I’ll grant you, of course, that anonymous reviews can sometimes be nasty.  We’ve all gotten infuriating reviews (and some of us have done stupid things as a result), but in my experience as an editor, truly inappropriate reviews are quite rare**.  (It’s actually not awful if some inappropriate reviews get written, as long as editors do their jobs and redact them.  That some editors don’t do this drives me up the wall.)

My reason for signing is also not that a more localized belief that my own reviews will be better if they’re open (independently of the science-wide argument in the previous paragraph).  I honestly don’t think that signing changes my behaviour, in the sense that it ensures that my reviews are constructive and polite.  I’m tempted to claim that I won’t write anything in a review that I wouldn’t say to an author’s face”, but if I’m honest, that isn’t quite true – which is why I carefully specified that I sign most of my reviews.  Occasionally, I remain anonymous precisely because I wouldn’t be comfortable saying something to an author’s face.  But: this is not because I’m worried that in saying it, I’m behaving badly.  Instead, it’s because I’m worried about an author behaving badly in response – whether active retaliation or just an unconscious bias against me when my own work is up for review.  I think the former is very rare; but the latter probably isn’t.  We all remember when our work is criticized; and it’s only human to let those memories sway your future judgement even if you’re intellectually resolved that it shouldn’t***.  My occasional anonymity lets me be as critical as I think is needed.  I still try to phrase that criticism as constructively as I can, of course, and most reviewers do the same.

This watch-out-for-retaliation logic suggests a likely effect of experience on review-signing strategy.   I’m tenured and thus impervious to the most serious kinds of damage; early-career researchers have a different cost-vs.-benefit situation and might wisely remain anonymous more often than I do.  I’d worry that initiatives to require signed review would reduce reviewer diversity by discouraging early-career researchers (who make terrific reviewers) – although the admirable commitment of many ECR to the open-science movement might suggest this is less of a problem than I think.

So if it isn’t to improve science in general, and isn’t to improve my own reviews, why do I sign?  It’s actually quite simple: so that authors who want to can contact me for fuller discussion of my review****.  Maybe what I’ve written is unclear.  Maybe the authors would like to try out a proposed fix on me before committing themselves to a full manuscript revision.  Maybe what I’ve written suggests something to the authors that neither they nor I thought of, and that they’d like to chat about.  If an exchange of emails can get a paper revised better, or in one step rather than a repeated series of submissions and new reviews, then it seems like everyone wins.

Peer review has two functions, and they’re quite different.  The first is gatekeeping – providing some meaning to the credential supplied when something appears in the peer-reviewed literature.  I don’t need to sign to provide that function. The second function, and the one I find I’m more interested in, is manuscript improvement.  Peer reviewers always make my science better – and as a reviewer, my ambition is always to do the same for another author.  Signing my reviews is one way I try to make my reviewing more effective, because it means the review itself can be the beginning rather than the end of a conversation about how a manuscript could be better.

Here’s the thing, though – I’m not sure it works.  In practice, few authors ever contact me.  I haven’t kept track, but it has to be less than 1 in 20. If so, then there seems to be little benefit to my signing.  That leaves only the costs.  I’ve figured this out, but I still (almost always) sign.  I guess I have a choice: I can think of myself as foolish, for sticking with a practice that cost-benefit analysis can’t support; or I can think of myself as selflessly tilting at a worthwhile windmill.  You can probably guess which way I lean.

© Stephen Heard (sheard@unb.ca) March 15, 2017

UPDATE: Here’s an older but very relevant post by Tom Webb, coming down on Meg’s side.  (And with a nice bonus Friends reference.)


*^There are even good arguments that even if signing helps reviewers write reviews better, this is offset by the fact that it leads authors to read and respond to them worse.  As Hope Jahren put it on Twitter: “As an Editor I always liked it best when reviewers did not sign. Once somebody’s name was real, it shifted focus from the content of the review and the MS itself [to] personalities: ‘well, s/he *would* say that, because s/he is/isn’t XYZ ..’ For a bad anonymous review authors would reply with ‘reviewer believes we said X and we really said Y’. For a bad identified review authors would reply with ‘Dr. So&So has always been opposed to the idea of Y’.  The former represented a much more productive process for the ultimate growth of the paper, I found.”

**^It’s an apparency thing.  The few nasty reviews that do exist get tweeted about, ranted about, shared with friends, and spread widely.  The huge majority of helpful ones get left on desktops.  When was the last time someone said to you “Hey, let me tell you about the really sensible and helpful thing a reviewer wrote about my paper”?

***^I’m disagreeing here, I think, with Jeremy Fox over at Dynamic Ecology, who thinks we needn’t worry about “disagreeing publicly with Dr. Famous”.  I’d like very much to agree with him, but can’t quite.

****^It’s been suggested to me that journals might not appreciate my opening the door to author-reviewer discussion outside the formal peer review system.  When I have my editor’s hat on, such a thing wouldn’t bother me – but in case it bothers other editors, I make my invitation to the authors explicit, right where I sign my review.  I’ve never been asked to take it out.

Advertisements

26 thoughts on “Why I sign (most of) my reviews

  1. Pingback: Why I don’t sign (most of) my reviews | Dynamic Ecology

  2. lizmartinsilverstone

    I completely agree about the contact aspect. Most people in my field (or at least the reviewers I’ve had) have all been open and signed their reviews, or contacted me to tell me it was them. I have frequently contacted them about comments they made, asking to clarify a point or if they had a reference for something, which I think is a great way of doing it.

    As a PhD student I haven’t reviewed a lot of papers, but I think I’m about 50/50 for whether or not I sign them. For me it depends on how critical I’m being, and who the authors are. Some people I know are find with me criticising them, while others I know could make my life extremely miserable, especially as I’m looking for post docs.

    I think the best way forward is to allow the option. In general, I think it’s good to sign it, but there are definitely reasons why it could be harmful for you to sign, especially as an early career scientist, and I don’t think we should be forced to if we think it wouldn’t be beneficial

    Liked by 1 person

    Reply
  3. Simon Leather (@EntoProf)

    I’m a non-signer, not because I wouldn’t say it to an author’s face but because I think it places an unconscious obligation on the author to say nice things about your paper if he/she gets asked to review one (and vice-versa of course).

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      Yes, Simon, this is a very sensible argument (Meg mentions it in her post); I can’t disagree. I guess I just feel that that in the big cost/benefit calculation, the risk of encouraging quid-pro-quo isn’t so large, compared with the benefit for manuscript improvement that I hope for (but apparently am not realizing) by signing. I don’t *think* that I personally deliver a more positive review in exchange for a positive one; but unconscious biases are called “unconscious” for a reason, so it’s possible!

      Like

      Reply
  4. JW

    As an author, I have only ever had one reviewer sign their review, but I did contact the reviewer and it improved that manuscript, plus led to a collaboration and another paper! So there’s an n=1 example of signing a review working out well for all parties. I’ve not signed any of my reviews, though, but I think my impulses are somewhat similar to yours, Steve, because I’ve been most tempted to sign reviews of papers that need a lot of help and I’m actually quite critical–where the data are not irredeemable, but there are major issues with analyses and writing. My main reason for not signing, though, is hesitance to (potentially) get entangled in someone else’s work (not so altruistic after all!), but given your experience I probably needn’t worry about that. In these cases, I haven’t been worried about the possibility of retribution, even as an early-career person, but I might feel differently if I were criticizing the work of more well-known people.

    Like

    Reply
  5. Jeremy Fox

    I just wanted to say that I have never in my life been contacted by an author who wanted to talk about a review, or heard of anyone else doing this. And as an author, I’ve never had any desire to contact reviewers to discuss their reviews of my work. It never occurred to me, and even now that it has I still have no desire to do it.

    You learn something new every day. Today I’ve learned is “Stephen Heard is an odd duck”. 😉

    Liked by 1 person

    Reply
      1. Jeremy Fox

        Re: “it does happen sometimes”, this once again raises the question of whether one should routinely do X in the hope that doing X will promote, or at least not prevent, unlikely positive effect Y. I’m recalling our debate about whether one should routinely wear one’s conference nametag when away from the conference center in the hope of promoting, or at least not preventing, the unlikely possibility of a spontaneous conversation with a student who wouldn’t otherwise have been able and willing to talk to you.

        Like

        Reply
  6. jeffollerton

    I mostly sign and I’ve certainly been approached by authors about my review a number of times; it once led to a collaboration when I suggested a completely different way of approaching and analysing the data set. I struggled with the ethics of that one for a while but ultimately decided it was ok (though I can see arguments against it).

    Like

    Reply
    1. ScientistSeesSquirrel Post author

      Interesting. That hasn’t happened to me, but I would struggle just like you with whether being added would be kosher or not – could be seen as the authors co-opting a reviewer, but as the journal would certainly re-review, I guess I’d not see that as an insurmountable problem. You should blog about that case and your decision!

      Like

      Reply
      1. jeffollerton

        Yes, perhaps when I have time… 🙂 Just to be clear, the original manuscript was rejected from the journal concerned and the re-written/analysed manuscript (with me as co-author) went to a different journal, so there was no conflict of interest in that sense.

        Like

        Reply
    2. Jeremy Fox

      More than once as a reviewer I have felt like I contributed enough to a paper I reviewed to have earned co-authorship had it been on offer. But I do think it would be a bad idea for reviewers to start signing reviews in hopes of being brought on board as co-authors, and for authors to start inviting reviewers on board. It creates a serious conflict of interest and creates really perverse incentives for reviewers. But I’m thinking about this in the abstract and don’t know the details of the specific instance in which you were involved Jeff, so I second Stephen’s interest in reading a blog post about your experience and why you ultimately decided it was ok ethically. If it wasn’t a matter of you being brought on as a co-author of the paper you were reviewing, but rather was a collaboration that led to a separate paper, I could see that possibly being ok.

      Like

      Reply
      1. jeffollerton

        Oh yes, I certainly don’t sign reviews in the hope of garnering papers! This was a matter of reviewing a manuscript where the authors had collected lot of interesting data but really didn’t know what to do with it, and the subsequent manuscript reflected that. In my review I suggested basically starting again from a different perspective and using a different set of analyses. So the final paper bore little relation to the original one.

        Like

        Reply
  7. jpschimel

    I went through a phase of signing my reviews believing that it was the noble thing to do. Then I hit a paper so bad, there was no way I was going to sign my name to a review that was largely a dissection of junk. So, then I realized that if I only sign the good reviews, I’m not being noble, I’m asking for pats on the head. So I stopped signing reviews as a matter of course. I do try to write all my reviews with the thought of “would I be comfortable standing behind my words as if I had signed the review?” But there are times when I feel I have to be critical of people I know; sometimes I would rather not have them know it was me.

    The reviews I sign now are the long, complex ones where I think there is something cool in the work but where there are parts I want to take issue with and argue about–but where I may not be at all sure I’m right. A) I figure with those it’s probably pretty obvious who wrote it, and B) those are where I am saying that I am happy to open up a discussion.

    That strategy has paid off for me fabulously. I got one of Stefano Manzoni’s first Ph.D. papers to review. The modeling was awesome, but I thought the soil biology a little naive. No surprise, really since he was a Ph.D. in a civil engineering Department. I wrote a 6 page review and noted that some of the N-mineralization dynamics he was analyzing might reflect spatial structuring in soil rather than different processes. I signed that review. A few months later, I got an e-mail from Stefano; it had a new paper attached. He’d picked up on those spatial ideas, written a new model, and I was invited to be a co-author on the new paper. He didn’t need to do that, but then Stefano is a true gentleman and a scholar. He’s also been a friend and valuable colleague since–that was the first of four papers we’ve co-authored together (plus a major NSF grant). So signing that review paid off in spades for me, and I hope he thinks for him as well.

    Liked by 1 person

    Reply
    1. jeffollerton

      I think that this comment captures one of the reasons why reviews should be signed; what you refer to as “junk” is a piece of work in which the authors(s) have invested considerable time and effort. They also have an emotional investment in the manuscript, as we all do.

      Even if you do not agree with the way in which the work has been undertaken, the analyses, the conclusions, the way it’s been written or its whole rationale, it is possible to write a highly critical review without dismissing it entirely, and by suggesting ways in which the work can be improved. As Stephen has said many times, this is arguably the most important function of peer review.

      Like

      Reply
      1. Chris Mebane

        That emotional investment is a good reason not to sign reviews, as a rule. As a rule though, rules seldom fit all situations, so occasionally it’s best to step around rules. Completely support all else. No one sets out to write a piece of junk article or a poor experiment, but sometimes they just turn out that way from the perspective of others. No one wants to hear it, even in the gentlest, most positive, forward looking way possible.

        Liked by 1 person

        Reply
    2. Chris Mebane

      OK, I just chimed in on the con side, but Josh makes a good point. Reviewers are often invited because of their closely related research, and at some point it’s self-evident who’s reviewing so why keep up the pretense? Probably the author of the closely related work suggested the reviewer in the first place, and as noted. Some of the comments on both the “sign” or “not” postings mentioned games reviewers play. Undoubtedly, but in my experience the vast majority are straightforward, collegial, mostly-constructive reviews following the Golden Rule: accept review invitations and complete reviews in the manner that you hope to receive. McPeek’s “Golden Rule of Reviewing” in AmNat is worth a periodic re-read (http://dx.doi.org/10.1086/598847)

      Like

      Reply
    3. ScientistSeesSquirrel Post author

      Interesting that you still make those exceptions, Josh – in a way, your strategy is almost the converse of mine. I do agree that my “allows discussion” rationale makes most sense for reviews that are complex or when I’m unsure I’m right about my comments. And I like your story about gaining a friend and colleague. Thanks for weighing in!

      Like

      Reply
  8. Staffan Lindgren

    I am a non-signer. Mostly because I feel less constrained for various reasons, some of which have been mentioned above. In the end, whether you sign or not, if you are critical in a constructive and non-condescending way, it shouldn’t matter. Condescending reviews are more likely to be dismissed by the author(s) than a constructive and thorough review. A number of years back, I wrote a dismissive review of a “junk” paper that I regret. It is not what you say, but how you say it that is important.

    Liked by 1 person

    Reply
  9. Pingback: Poll: have you ever contacted a reviewer about a review before responding to it? Or received such contact? | Dynamic Ecology

  10. Pingback: To sign or not to sign: what the Replies taught me | Scientist Sees Squirrel

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s