Even updateder update: Jeremy Berg has posted an analysis of application scoring for the October 2010 Council pool (654 R01s) at the Feedback Loop, with similar trends in Approach and Significance.
Update: Jeremy Berg has posted similar analyses of Approach and Innovation scores at the Feedback Loop … and now, regression analysis results, too!
My NIGMS Feedback Loop listserv alerted me to Jeremy Berg’s assessment Model Organisms and the Significance of Significance. Not much on model organisms (an interesting comment by Whimple – update: and others now), but Dr. Berg notes that:
To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. [he shows a plot, too]
As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.
Hmm. Not too surprising. Research is not likely to have much impact if it is not both significant (meaningful) and well designed/planned. I realized on reading his post that I do indeed tend to discount the scores (and, to some extent, the comments) under the other criteria and focus on the overall impact bullets plus Significance and Approach when reviewing Summary Statements.
Likelihood for the project to exert a sustained, powerful influence on the research field(s) involved
- Likelihood (i.e., probability) is primarily derived from the investigator(s), approach and environment criteria
- Sustained powerful influence is primarily derived from the significance and innovation criteria
Though I still focus on assessment of Significance and Approach in the review …
I’ll be interested to see if these data change with the just-completed reviews of the first short-format applications submitted during Cycle 1. If anything, I would expect them to become more tightly correlated, which is I’m sure what Toni Scarpa hopes as well. Then again, the Summary Statements from this round that I’ve already read invariably note something to the effect that details are lacking (in approach), so we’ll see.
(and, after ignoring the blogosphere for a few weeks due to travels & grant overload, I just thought to check, and, yes, DrugMonkey covered this as well … but in case there’s anyone here but not there who might be interested in the NIGMS Feedback …)