This is a guest post by Jeannette Whitton, Group Chair for Group 1503 (Ecology & Evolution) of the Natural Sciences and Engineering Research Council (Canada) (and Professor of Botany, University of British Columbia). She has extensive experience with the review and evaluation of NSERC Discovery Grants (among other things!) While Jeannette writes here specifically about reviewing proposals for Discovery Grants, much of her advice will serve you well in reviewing other kinds of grants, or grants for other agencies*. It will also serve you well in writing grants – because if you know what reviewers and evaluation panels are looking for, you can deliver just what they need. Dig in!
Some weeks ago, you graciously agreed to review an NSERC Discovery Grant (DG) proposal, or possibly two or three**, which makes you an awesome person, especially in 2020. Because of confidentiality issues, we don’t get much training with reviewing grants – but just as for manuscript reviews, it takes time and care to provide a thoughtful grant review. How I review DGs changed after I served on the evaluation panel and got to see what was most useful, so I thought I would write down some thoughts about what to focus on. I hope this helps those who are new to NSERC DG reviews – or to reviewing grants more generally. Comments are most welcome!
A mindset for DG External Reviewers
Your job as a grant reviewer is to help the evaluation panel members see the nuances in the applicant’s accomplishments and their proposed research. Evaluation group members assess grants well beyond their own narrow research expertise. You will have more expertise with at least some aspects of the proposal or the subdiscipline than the panel members, although they’ll have greater familiarity with how the proposal fits into NSERC’s assessment criteria. Your assessment will ideally complete and complement their assessment. It can be helpful to state your expertise in your review, or to identify and focus on areas that you are most comfortable assessing.
Do this first, well before you need to review the proposal
I’ve never heard anyone lavish praise on NSERC’s online system – not once. Familiarity only helps a little. In order to separate your feelings about the web portal from your feelings about the grants you have to review, I urge you to log in and download the files you will need early, not on the day you plan to complete the review. While you’re on the portal, look over the form you’ll need to fill out. Note that the reviews are structured! There are multiple prompts with boxes to fill, which cover each of NSERC’s three assessment criteria: the excellence of the researcher, the merit of the proposal, and the contributions to training of highly qualified personnel (HQP). It may help you to copy over the prompts and organize your notes under these headings. You could organize your notes and save your work on the portal, but that seems like it would be more of a pain than cutting and pasting from a text editor.
Things you don’t need to do (and/or really should not do)
- You don’t need to summarize the application for the panel. The five grant panel members assigned to evaluate the proposal typically read the external reviews after they’ve done their own assessment, so they already know the content.
- Don’t focus on lists and numbers of publications or other contributions, both because those are summaries, but also because your emphasis should be on the impact of contributions, not on simple summary metrics. If a work is well cited, that’s great, but it is more helpful to comment on how the work is impacting the field. Papers with few citations can still be high in impact.
- Do not undertake your own research on the applicant – don’t check out their website, google them, or otherwise check on their record. Evaluate only the materials in the application, using your expert knowledge.
- You can use your prior knowledge of the applicant and their work to inform your assessment, but don’t extend beyond the 6-year window relevant to the application***, for example to describe their important work from 2003****.
- Don’t analyze the budget. Discovery grants are meant to fund research programs, not specific projects. If the program is funded, the applicant will have to figure out how to allocate the funds awarded to best move their program forward.
- Generally, don’t qualify your assessment on career stage, teaching load, or other conditions of the applicant’s appointment. The proposal is the proposal!
- Don’t be mean. The external reviews are sent to the applicant, and are typically the only written feedback they get. Be kind in your critiques.
- Don’t worry too much about the deadline NSERC gives for your review in early January. It’s great if you can submit your review on time – it will greatly help NSERC staff manage their workload – but if you can’t, aim to submit as soon as you can. The life sciences panels meet in mid-February, and reviews will get passed along to the panel as they trickle in. This means you should go sledding on New Year’s Eve, if at all possible.
What is most helpful
Below I list some prompting questions and a few tips for each of the three areas you are asked to assess. You don’t need to comment on all of these; you can highlight the areas that particularly struck you, or for which you have particular insight. Overall, my feeling is that if you have to prioritize, it’s best to focus on merit of the proposal, because that is likely to be the area where your specific expertise was sought, and is most valuable.
Excellence of the Researcher:
- How does their work – not just the current proposal – relate to important questions being addressed in the field? What is the potential for the applicant’s work to contribute to longer term objectives and broader goals within the applicant’s program, and more generally, in the discipline?
- Do they publish in venues that are important within your subdiscipline, but perhaps less well known to non-specialists? Highlight this.
- Does their work influence policy, management, industry? Comment on the importance of these impacts.
- If you have time, comment on the sample contributions (the papers submitted with the proposal). What are the major insights from these? What is the likely scope of their impact?
- If the applicant lists leadership contributions, help the committee understand how these are valued, and how much work is involved.
- Comment on the applicant’s record relative to standards for the subdiscipline. This isn’t about the number of papers and talks, but about the impact that the work is having.
- Comment on the impact of venues in which the applicant has given talks, and on other forms of impact, such as outreach, media engagement, writing useful code, etc…
Merit of the Proposal
- Are you enthusiastic about the work being proposed? What do you look forward to hearing about in the future?
- What makes their research program unique and valuable?
- Has the applicant presented a clear outline that allows you to understand what they will do, who will be involved, and what the expected outcomes are?
- Do you have concerns about the study system, sampling, or methods being proposed? If you have suggestions for improvement, how deep are they? Are these oversights, differences of opinion, or misconceptions by the applicant that threaten the value of the work proposed?
- Is the work feasible as presented? Are the risks reasonable? Is there value in doing the proposed projects regardless of the specific outcome?
Contributions to training of highly qualified personnel (HQP)
The HQP criterion can be difficult for external reviewers to make impactful comments on. It gets easier to assess this criterion when you’ve evaluated a number of proposals. Nonetheless, it can be helpful if you can comment on how the training program compares with others in the research area.
- Is the applicant providing opportunities for HQP to advance their careers? This could take many forms and involve HQP at many career stages.
- Are the projects for HQP well-planned and appropriate for the HQP role (i.e., B.Sc., M.Sc., Ph.D., PDF)?
- Is there evidence that past HQP have benefited from their training?
Your work as a DG reviewer is highly valued by the members of the evaluation group, and it is also passed along to the applicants, whether they are funded or not. This underscores for me the importance of being supportive and helpful, even while providing a critique. As a reviewer, you’re first and foremost doing a service for your colleagues. Do unto them as you’d surely want them to do unto you!
Finally, be aware that if you do a great job of reviewing DGs, one of the panelists is likely to say, “we should recruit them for the panel”. Of course, this shouldn’t be a reason not to do a great job! There are worse fates than being sought after.
© Jeannette Whitton December 28, 2020
Image: Discovery Grant rating form (excerpt), from the NSERC Peer Review Manual (which is written for evaluation panel members, but contains extremely useful information if you’re reviewing a grant – and even more useful information if you’re writing one.
*^Whatever agency or program you’re reviewing for, check to see what specific instructions they might have for reviewers. You’ll see in this post that NSERC has some particular requests of its reviewers; other agencies might prefer you to focus on different elements of the proposal or pass different kinds of judgment. But that doesn’t mean this post isn’t useful for non-Discovery Grant reviewers. You can see the most NSERC-specific bits of advice here as suggestion to ask yourself what the corresponding advice would be for agency X.
**^You should not be asked to review more than 3 grants in a year, but requests are not coordinated between Evaluation Groups. Feel free to decline some, but consider prioritizing reviews for early career researchers.
***^This is more like a 7-year window. For grants submitted in Fall 2020, the window normally includes 2014-2020, but extensions to this window can now be included – basically, restrict your comments to the time frame included in the application materials.
****^This is important because NSERC takes a ‘recent accomplishments’ approach to assessing applicants, which helps reduce the advantage of well-established researchers with long records. Exceptions are allowed if that 2003 paper still impacting the field today.