NIH Updates on Scoring & Review Criteria

Update: NIAID, per usual, has an excellent table comparing the old and new review processes.

Here is the official updated implementation timeline for enhancing peer review:

January 2009 Due Dates
Early Stage Investigator and New Investigator Policy
New NIH Policy on Resubmissions

May 2009 Review Meetings
9-Point Scoring System
Enhanced Review Criteria
Formatted Reviewer Critiques
Scoring of Individual Review Criteria
Clustering of New Investigator Applications During Review

January 2010 Due Dates
Shorter Applications for R01s and Other Mechanisms
Restructured Applications to Align with Review Criteria

The enhanced scoring procedures will use a whole-number, 9-point scale (1=exceptional; 9=poor), with percentiles starting from scratch from the May 2009 cycle on. Each application, including those not discussed, receives a score for all 5 core review criteria (see below). For discussed applications:

Before the review meeting, each reviewer and discussant assigned to an application will give a preliminary impact score for that application. The preliminary impact scores will be used to determine which applications will be discussed. For each application that is discussed, a final impact score will be given by each eligible committee member (without conflicts of interest). Each member’s impact score will reflect his/her evaluation of the overall impact that the project is likely to have on the research field(s) involved, rather than a weighted average applied to the reviewer’s scores given to each criterion (see above).

The overall impact score for each discussed application will be determined by calculating the mean score from all the eligible members’ impact scores, and multiplying the average by 10; the overall impact score will be reported on the summary statement. Thus, the 81 possible overall impact scores will range from 10 – 90. (Overall impact scores will not be reported for applications that are not discussed.)

Funding Decisions. The new scoring system may produce more applications with identical scores (“tie” scores). Thus, other important factors, such as mission relevance and portfolio balance, will be considered in making funding decisions when grant applications are considered essentially equivalent on overall impact, based on reviewer ratings.

And about those enhanced review criteria

Overall Impact. Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following 5 core review criteria, and additional review criteria (as applicable for the project proposed).

Core Review Criteria. Reviewers will consider each of the 5 review criteria below in the determination of scientific and technical merit, and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may be essential to advance a field.

Significance. Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

Investigator(s). Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?

Innovation. Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

Approach. Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?
If the project involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed?

Environment. Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?

1 Comment »

  1. […] These Challenge Grant applications will be reviewed using the new criteria and scoring procedures. […]

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: