More on the Impact-Criterion Score Correlation

This time by Sally Rockey on Rock Talk.

Jeremy Berg introduced the concept of correlating overall impact score with the individual criterion scores, first using NIGMS and then NIH-wide data.

Based on the 32,546 applications (of 54,727 submitted) that received overall impact scores in FY10, OER played with the numbers a bit more but came up with the same conclusions: Approach and then Significance drive Overall Impact scores.

For applications receiving numerical impact scores (about 60% of the total), we used multiple regression to create a descriptive model to predict impact scores using the applications’ criterion scores, while attempting to control for ten different “institutional” factors (e.g., whether the application was new, a renewal, or a resubmission). In the model, scores for the approach criterion had the largest regression weight, followed by criterion scores for significance, innovation, investigator, and environment. The same pattern of results was observed across multiple rounds of peer review and institute funding decisions.

She also notes, as can be seen in her figure, that scores for Approach showed the widest range, followed by Significance.

So, the work you propose doing better be important … and, more importantly, better be done right.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: