CSR Scoring Recalibration

NIGMS has kindly publicly confirmed that CSR is recalibrating percentiles, having pushed SRGs to enforce the spreading of scores when reviewing Cycle III applications this past February and March.

Last December, CSR posted a new graphic illustrating how the 9-point scoring should be interpreted:

CSR scoring

Colleagues and alert readers have been sharing their experiences, both on the giving (SRG) and receiving (PI) end. Reviewers were told to spread the scores, PIs were puzzled (often pleasantly) about their score/percentile. For example, in Cycle III, a score of 30 at the 7th percentile … compared with a score of 19 and 7th percentile in Cycle II … both going to NCI for consideration (do not know if they are the same SRGs). I have heard about a lot more scores in the 40s and 50s this cycle, generally described as in the Summary of Discussion as significant but with moderate weaknesses (which I suspect were being scored in the 20s & 30s last year). Just an FYI for those trying to work out their scores/percentiles.

About these ads

7 Comments »

  1. Left this comment at Loop:

    The announcement from CSR is confusing. They say that bases will be recalculated “for a number of permanent study sections and Special Emphasis Panels (SEPs)”. But then they say that new percentiles will be calculated “for applications that are reviewed in SEPs this round and should be percentiled”.

    So is it correct that no recalculation of percentiles occurred for grants reviewed in standing study sections? Is it correct that this was nothing more than a recalibration of the “all-CSR” base for non-recurring SEPs?

    • S said

      Incorrectte. Percentiles were also recalibrated for some CSR standing study sections. I am not sure if it was all. I think it depends on how well the study section had been spreading the scores in the past.

      • What is the evidence for this conclusion?

      • S said

        All knowing omniscience…plus discussions with SS members, SROs and POs. I do think that it is a big secret.

  2. laghs said

    I am very confused about the whole recalibration thing. My R21 was reviewed this month by an SEP, and received a score of 30 at the 18th percentile. I assume the same NIH-wide base would be applied to all applications from this cycle. But if so, how could someone got a score of 30 at the 7th percentile (while mine was only 18th)?

    • writedit said

      If your SEP was at an IC (versus CSR), this doesn’t apply. Otherwise, the SRGs and SEPs using recalibrated percentiles start from scratch, so the percentile reflects ranking among applications reviewed at that one study section meeting. The other example is an R01 and a different study section, and percentiles are linked to the review group and the mechanism. It could be your area of science had a lot more competition this cycle, or rather, that the 7th percentile example did not have much competition; your 18th percentile seems more appropriate, given the new scoring schema above, than a 7th percentile ranking. Of course, this is already a rather confusing, stressful FY to now have the scoring metrics rejiggered as well.

      • laghs said

        Thanks. This clarified things a lot. Yes, mine was reviewed by an SEP from NCI. So I guess it was not impacted at all by this recalibration exercise. I am going to submit an A1 for the next cycle. But looks like the NCI success rate for R21 is even lower than R01, so it is probably going to be a very big challenge bringing the score down to the 7th percentile.payline (2012, who knows about 2013?) or lower.

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 273 other followers

%d bloggers like this: