Recommendations for Improving NRSA Fellowship Review

Please provide your comments on the proposed changes to NRSA fellowship review criteria:

The National Science Policy Network (NSPN), an organization of over 650 early-career scientists, is highly invested in assuring equitable review of fellowship applications to foster the growth of a scientific community that better leverages our varied perspectives in science. We are an organization dedicated to engaging early career scientists in policy-making while fostering an inclusive, diverse, and equitable scientific community, and we appreciate the opportunity to offer input on the proposed changes to merit review criteria for predoctoral fellowships via this Request for Information (RFI) on Recommendations for Improving Kirschstein National Research Service Award (NRSA) Fellowship Review.

NSPN largely supports the proposal to consolidate the current five criteria into three criteria that focus on the trainee’s potential for growth and the merit of their science. We anticipate that these changes will create a more level playing field between meritorious applicants from various institutional backgrounds. We also believe this will deemphasize the star power of senior-level sponsors and lead to a more equitable review of applicants sponsored by early career faculty.

These updates, while a step in the right direction towards diversifying the biomedical research workforce, still do not address a central issue within the F31 application process: the lengthy timeline associated with an F31 application. The review process for research project grants (RPG) is fairly similar to that for the NRSAs. However, faculty submitting RPGs have several years at an institution to review and resubmit these applications. This contrasts with the comparatively short timeline that trainees have to apply for a fellowship. Currently, graduate student applicants are expected to be between their second and third year when submitting an F grant application, such that they leave adequate time to enact their training plan. These applicants, if fortunate in their initial submission timing, can resubmit one time within this window. Study section comments are often returned to applicants with as little as two weeks before the next submission deadline. With internal institutional review deadlines, this leaves less than one week for an applicant to address reviewer comments and concerns from the initial submission. Thus, even though applicants technically have a chance to apply every four months, current practices force many applicants to lose precious training time while they wait a full eight months before resubmitting if they can resubmit at all. This issue could be addressed by expediting the meeting of study sections. Study sections often do not meet until two months after the application deadline. Reducing this time would accelerate the resubmission timeline for applicants, thus allowing more time to refine their proposals and training plans, and resulting in more opportunities to apply.

Moreover, we believe that increasing opportunities to resubmit could further broaden the backgrounds of those who receive NRSA. Meritorious applicants from less prestigious institutions often have fewer grant support resources and institutional knowledge available to them than their highest research activity counterparts. Therefore, applications from less traditional backgrounds stand to benefit most from reviewer feedback and the resubmission process.

Finally, we would like to draw attention to the persistent issue that reviewer scores often contrast reviewer-stated strengths and weaknesses. A reviewer may assign a poor score to an application even though they cited significant strengths. Or worse, they may leave little to no comment at all. Such occurrences may seem biased and arbitrary, which only operates to exclude potential excellent scientists from further engagement with research careers. Therefore, with the proposed updates to the evaluation criteria, we strongly encourage more transparency for reviewer scoring. This could be achieved by making use of the system already in place for unscored items. If an item is unsatisfactory, it might be stated clearly why it is unsatisfactory. Requiring statements that reflect the wording of the impact score rubric is one way that scoring transparency could be increased while offering additional helpful feedback to the applicant.

Please provide your comments on the proposed changes to the NRSA application instructions and materials:

We appreciate updates to the application instructions to better serve applicants from diverse backgrounds. While we agree that disclosure of grades should not be mandatory, we believe it is an oversight to disallow the consideration of an applicant's academic record entirely. While some applicants may have a weaker academic record due to socioeconomic (or similar) hardships and cannot be reasonably compared to applicants without this background, grades can still be a significant marker of understanding an applicant’s success in the context of their available opportunities. Applicant who maintains a pristine academic record should be able to use their grades as evidence of their hard work and potential for achievement, especially in the absence of other research opportunities. We believe that the proposed Statement of Special Circumstance can address the equity issue often attributed to the consideration of grades by allowing applicants who wish to withhold their grades to explain extenuating circumstances or hardships, should they desire. Alternatively, students could also lobby for consideration of their grades to demonstrate how they succeeded in their available opportunities. Removing data points from consideration removes the nuance required of considering applicants in the context of their background. 

We strongly support the inclusion of the Statement of Special Circumstances. The opportunity to describe events that impacted an applicant’s academic journey is expected to promote the inclusion of fellows from non-traditional backgrounds. However, we believe further clarity is needed regarding the role of this statement in the merit review process. We would appreciate the opportunity to comment on who will read the statement of special circumstances and how it may alter an applicant’s impact score. Without this specification, the Statement of Special Circumstances may be weaponized against an applicant instead of aiding them where necessary.

Finally, we believe that reference letters will need to be further adjusted to have the desired impact of broadening participation. As it stands, the F31 application requires three reference letters, excluding the sponsor and others associated with the proposed project. This is already a heavy lift for applicants who have to request letters from overburdened researchers who are no longer immediately beholden to the applicant. These letters of reference are an additional burden on students from non-traditional backgrounds, who may have little research experience beyond their graduate program and relevant experience in other areas. As it stands, question three of the proposed instructions implies that these letters must come from someone who oversaw the applicant in research experience. However, there are qualities of an excellent scientist that are also observable outside of a strict research context. Thus, the reference letter instructions should be further adjusted to explicitly welcome reference letters from non-research contexts, such as academic advisors and previous employers who were able to witness trainees succeed in their available opportunities.

We thank the NIH and Center for Scientific Review for their work on reforming the fellowship review criteria process to make it more accessible and equitable for participants from diverse backgrounds. We look forward to further stakeholder engagement as more details are developed and through the implementation of these new criteria.