Feedback on NIH Scoring

Even updateder update: Jeremy Berg has posted an analysis of application scoring for the October 2010 Council pool (654 R01s) at the Feedback Loop, with similar trends in Approach and Significance.

Update: Jeremy Berg has posted similar analyses of Approach and Innovation scores at the Feedback Loop … and now, regression analysis results, too!

My NIGMS Feedback Loop listserv alerted me to Jeremy Berg’s assessment Model Organisms and the Significance of Significance. Not much on model organisms (an interesting comment by Whimpleupdate: and others now), but Dr. Berg notes that:

To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. [he shows a plot, too]

As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.

Hmm. Not too surprising. Research is not likely to have much impact if it is not both significant (meaningful) and well designed/planned. I realized on reading his post that I do indeed tend to discount the scores (and, to some extent, the comments) under the other criteria and focus on the overall impact bullets plus Significance and Approach when reviewing Summary Statements.

I actually like this definition of Overall Impact from Sally Amero’s presentation on peer review at the June 2010 NIH Regional Grant Seminar:

Likelihood for the project to exert a sustained, powerful influence on the research field(s) involved

  • Likelihood (i.e., probability) is primarily derived from the investigator(s), approach and environment criteria
  • Sustained powerful influence is primarily derived from the significance and innovation criteria

Though I still focus on assessment of Significance and Approach in the review …

I’ll be interested to see if these data change with the just-completed reviews of the first short-format applications submitted during Cycle 1. If anything, I would expect them to become more tightly correlated, which is I’m sure what Toni Scarpa hopes as well. Then again, the Summary Statements from this round that I’ve already read invariably note something to the effect that details are lacking (in approach), so we’ll see.

(and, after ignoring the blogosphere for a few weeks due to travels & grant overload, I just thought to check, and, yes, DrugMonkey covered this as well … but in case there’s anyone here but not there who might be interested in the NIGMS Feedback …)



  1. DrugMonkey said

    I really hope Director Berg continues to post these analyses so we can see if reviewer behavior is shifting in the direction they would like. His graphs also make a nice counter to a prior NIAID dataset from a single study section which showed the distribution peaks at the round integer scores.

  2. writedit said

    As noted above (but to alert those of you monitoring comments), Jeremy Berg has posted similar analyses of Approach and Innovation scores at the NIGMS Feedback Loop.

  3. Neuro-conservative said

    Why do I keep getting an error when I try to got to the Feedback Loop?

  4. writedit said

    And now, new and improved with regression analysis results! Boy, these 360 summary statements are getting a lot of scrutiny …

  5. writedit said

    Even better, as noted above, analyses of scores from the 654 R01s reviewed for the October 2010 Council round.

  6. […] Based on the interest in this analysis reflected here and on other blogs, including DrugMonkey and Medical Writing, Editing & Grantsmanship , I want to provide some additional aspects of this […]

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: