How to review an NSERC Discovery Grant

This is a guest post by Jeannette Whitton, Group Chair for Group 1503 (Ecology & Evolution) of the Natural Sciences and Engineering Research Council (Canada) (and Professor of Botany, University of British Columbia). She has extensive experience with the review and evaluation of NSERC Discovery Grants (among other things!) While Jeannette writes here specifically about reviewing proposals for Discovery Grants, much of her advice will serve you well in reviewing other kinds of grants, or grants for other agencies*. It will also serve you well in writing grants – because if you know what reviewers and evaluation panels are looking for, you can deliver just what they need.  Dig in!

Some weeks ago, you graciously agreed to review an NSERC Discovery Grant (DG) proposal, or possibly two or three**, which makes you an awesome person, especially in 2020. Because of confidentiality issues, we don’t get much training with reviewing grants – but just as for manuscript reviews, it takes time and care to provide a thoughtful grant review. How I review DGs changed after I served on the evaluation panel and got to see what was most useful, so I thought I would write down some thoughts about what to focus on. I hope this helps those who are new to NSERC DG reviews – or to reviewing grants more generally.  Comments are most welcome!

A mindset for DG External Reviewers

Your job as a grant reviewer is to help the evaluation panel members see the nuances in the applicant’s accomplishments and their proposed research. Evaluation group members assess grants well beyond their own narrow research expertise. You will have more expertise with at least some aspects of the proposal or the subdiscipline than the panel members, although they’ll have greater familiarity with how the proposal fits into NSERC’s assessment criteria. Your assessment will ideally complete and complement their assessment. It can be helpful to state your expertise in your review, or to identify and focus on areas that you are most comfortable assessing.

Do this first, well before you need to review the proposal

I’ve never heard anyone lavish praise on NSERC’s online system – not once. Familiarity only helps a little. In order to separate your feelings about the web portal from your feelings about the grants you have to review, I urge you to log in and download the files you will need early, not on the day you plan to complete the review. While you’re on the portal, look over the form you’ll need to fill out. Note that the reviews are structured! There are multiple prompts with boxes to fill, which cover each of NSERC’s three assessment criteria: the excellence of the researcher, the merit of the proposal, and the contributions to training of highly qualified personnel (HQP). It may help you to copy over the prompts and organize your notes under these headings. You could organize your notes and save your work on the portal, but that seems like it would be more of a pain than cutting and pasting from a text editor.

Things you don’t need to do (and/or really should not do)

  • You don’t need to summarize the application for the panel. The five grant panel members assigned to evaluate the proposal typically read the external reviews after they’ve done their own assessment, so they already know the content.
  • Don’t focus on lists and numbers of publications or other contributions, both because those are summaries, but also because your emphasis should be on the impact of contributions, not on simple summary metrics. If a work is well cited, that’s great, but it is more helpful to comment on how the work is impacting the field. Papers with few citations can still be high in impact.
  • Do not undertake your own research on the applicant – don’t check out their website, google them, or otherwise check on their record. Evaluate only the materials in the application, using your expert knowledge.
  • You can use your prior knowledge of the applicant and their work to inform your assessment, but don’t extend beyond the 6-year window relevant to the application***, for example to describe their important work from 2003****.
  • Don’t analyze the budget. Discovery grants are meant to fund research programs, not specific projects. If the program is funded, the applicant will have to figure out how to allocate the funds awarded to best move their program forward.
  • Generally, don’t qualify your assessment on career stage, teaching load, or other conditions of the applicant’s appointment. The proposal is the proposal!
  • Don’t be mean. The external reviews are sent to the applicant, and are typically the only written feedback they get. Be kind in your critiques.
  • Don’t worry too much about the deadline NSERC gives for your review in early January. It’s great if you can submit your review on time – it will greatly help NSERC staff manage their workload – but if you can’t, aim to submit as soon as you can. The life sciences panels meet in mid-February, and reviews will get passed along to the panel as they trickle in. This means you should go sledding on New Year’s Eve, if at all possible.

What is most helpful

Below I list some prompting questions and a few tips for each of the three areas you are asked to assess. You don’t need to comment on all of these; you can highlight the areas that particularly struck you, or for which you have particular insight. Overall, my feeling is that if you have to prioritize, it’s best to focus on merit of the proposal, because that is likely to be the area where your specific expertise was sought, and is most valuable.

Excellence of the Researcher:

  • How does their work – not just the current proposal – relate to important questions being addressed in the field? What is the potential for the applicant’s work to contribute to longer term objectives and broader goals within the applicant’s program, and more generally, in the discipline?
  • Do they publish in venues that are important within your subdiscipline, but perhaps less well known to non-specialists? Highlight this.
  • Does their work influence policy, management, industry? Comment on the importance of these impacts.
  • If you have time, comment on the sample contributions (the papers submitted with the proposal). What are the major insights from these? What is the likely scope of their impact?
  • If the applicant lists leadership contributions, help the committee understand how these are valued, and how much work is involved.
  • Comment on the applicant’s record relative to standards for the subdiscipline. This isn’t about the number of papers and talks, but about the impact that the work is having.
  • Comment on the impact of venues in which the applicant has given talks, and on other forms of impact, such as outreach, media engagement, writing useful code, etc…

Merit of the Proposal

  • Are you enthusiastic about the work being proposed? What do you look forward to hearing about in the future?
  • What makes their research program unique and valuable?
  • Has the applicant presented a clear outline that allows you to understand what they will do, who will be involved, and what the expected outcomes are?
  • Do you have concerns about the study system, sampling, or methods being proposed? If you have suggestions for improvement, how deep are they? Are these oversights, differences of opinion, or misconceptions by the applicant that threaten the value of the work proposed?
  • Is the work feasible as presented? Are the risks reasonable? Is there value in doing the proposed projects regardless of the specific outcome?

Contributions to training of highly qualified personnel (HQP)

The HQP criterion can be difficult for external reviewers to make impactful comments on. It gets easier to assess this criterion when you’ve evaluated a number of proposals. Nonetheless, it can be helpful if you can comment on how the training program compares with others in the research area.

  • Is the applicant providing opportunities for HQP to advance their careers? This could take many forms and involve HQP at many career stages.
  • Are the projects for HQP well-planned and appropriate for the HQP role (i.e., B.Sc., M.Sc., Ph.D., PDF)?
  • Is there evidence that past HQP have benefited from their training?

Your work as a DG reviewer is highly valued by the members of the evaluation group, and it is also passed along to the applicants, whether they are funded or not. This underscores for me the importance of being supportive and helpful, even while providing a critique. As a reviewer, you’re first and foremost doing a service for your colleagues. Do unto them as you’d surely want them to do unto you!

Finally, be aware that if you do a great job of reviewing DGs, one of the panelists is likely to say, “we should recruit them for the panel”. Of course, this shouldn’t be a reason not to do a great job! There are worse fates than being sought after.

© Jeannette Whitton  December 28, 2020

Image: Discovery Grant rating form (excerpt), from the NSERC Peer Review Manual (which is written for evaluation panel members, but contains extremely useful information if you’re reviewing a grant – and even more useful information if you’re writing one.


*^Whatever agency or program you’re reviewing for, check to see what specific instructions they might have for reviewers. You’ll see in this post that NSERC has some particular requests of its reviewers; other agencies might prefer you to focus on different elements of the proposal or pass different kinds of judgment. But that doesn’t mean this post isn’t useful for non-Discovery Grant reviewers. You can see the most NSERC-specific bits of advice here as suggestion to ask yourself what the corresponding advice would be for agency X.

**^You should not be asked to review more than 3 grants in a year, but requests are not coordinated between Evaluation Groups. Feel free to decline some, but consider prioritizing reviews for early career researchers.

***^This is more like a 7-year window. For grants submitted in Fall 2020, the window normally includes 2014-2020, but extensions to this window can now be included – basically, restrict your comments to the time frame included in the application materials.

****^This is important because NSERC takes a ‘recent accomplishments’ approach to assessing applicants, which helps reduce the advantage of well-established researchers with long records. Exceptions are allowed if that 2003 paper still impacting the field today.

 

5 thoughts on “How to review an NSERC Discovery Grant

  1. Tony Diamond

    Thank you for this very useful contribution, particularly as I am beginning to attack my own reviewing for NSERC right now! One thing I always struggle with, and would appreciate more help on, is assessing the impact of a researcher’s work, if NOT relying on publication venues and citation rates? Unless the individual is very well known – which usually means established in the field – how can their impact be assessed other than through citation rates? I agree these can be misleading, but what alternatives do you suggest?

    Like

    Reply
  2. Jeannette Whitton

    Great question, Tony. It’s difficult to give a simple answer for this, in part because your ability to assess impact depends upon your relationship to the content of the proposal. Is your expertise related to the group of organisms, the subdiscipline, the type of question, the methodological approach, etc., all of which affect a reviewer’s ability to provide value added comments on the proposal, including the impact of past contributions. Also, I should say that numbers of citations and venues CAN BE evidence of impact, but on their own, they don’t tell the full story. One example would be a paper or a set of papers that will allow the researcher to build impact over time – laying the groundwork for what they propose to do next. That’s a case where the impact as related by metrics might be low, but the work is nonetheless critical. In my mind, that kind of work can reflect favourably on the excellence of the researcher, if it demonstrates an understanding of what they need to build a new system, or a new way of approaching a problem.

    One tool that you can use is ‘the grid’ of merit indicators, used by the committee to guide the assignment of scores (https://www.nserc-crsng.gc.ca/_doc/Professors-Professeurs/DG_Merit_Indicators_eng.pdf). This can help you get a sense of what differentiates an applicant that scores very string versus strong, for example. Gavin Simpson, a recent Evaluation Group member on 1503 (Ecology and Evolution), wrote a solid blog post about his experience assessing grants. That’s here: https://fromthebottomoftheheap.net/2020/02/26/what-three-years-evaluating-discovery-grants-taught-me/ , and might also be useful.

    It falls to the applicant to describe the impact of their work. As an external, you can take what they include at face value if you have no additional insight, but in some cases, you may be able to provide context. For example, you might be able to say whether the work done by the applicant is important within a narrow area, or a broader field. If narrow, you might be well placed to comment on whether they are making average or above average contributions. For example, within a field like systematics, the scope of direct impact might be narrow (impacting our understanding of relationships within this group), but still leading within that discipline, for example by applying new tools or methods, or just being super-solid (as reflected by their sample and most important contributions sections, but also by publishing in strong journals within the field). They might achieve broader impacts (moving them up the grid) by resolving relationships in groups of conservation concern, or with economic impact, or through outreach activities that reflect recognition of their expertise. They should provide the information to assess impact, but you may be able to support their statements (or in some cases question these).

    Other insights you might have relate to what your overlap with the applicant’s work is. For example, if an applicant co-organized a symposium at conference that you typically attend, you might be able to say to what extent the awarding of symposia is indicative of leadership or impact (is there competition for these symposium slots). You might be able to comment on whether they are key participants in an emerging area (e.g. through working group participation, etc.). Some of this might repeat or pull together threads in the application, but ideally you can use your expertise to provide a value-added assessment on at least some aspects of their application.

    Like

    Reply
  3. Manu Saunders

    Thanks, very helpful post and wish there were more posts like this that could be useful for ECRs new to the grant applicant & reviewing context. (I’m not experiencecd with NSERC, only experience is with UK and Australia grant rounds, so just talking general big grant applications here). Great advice! But I would question the point advising “don’t check out their website, google them, or otherwise check on their record”. If you’re a reviewer in another country or from non-academic institutions, sometimes these sources are extermely helpful to understand the researcher’s history and context and capacity to deliver the grant outcomes (and I would add Twitter here, unpopular opinion as it probably is).

    Liked by 1 person

    Reply
    1. ScientistSeesSquirrel Post author

      Manu, glad you appreciated this! As for sleuthing: I understand the impulse. But it’s explicitly disallowed by NSERC because it easily leads to inequities in assessment – how can you assure that applicants are given equal amounts of outside-the-grant sleuthing? Without it, it’s an even playing field.

      Liked by 2 people

      Reply

Comment on this post:

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.