No.
Update: Emaderton3’s comment below includes this formula for converting the new impact score into the former priority score: [5 * (new impact score – 10)] + 100 = old priority score
e.g., [5 * (80 – 10)] + 100 = 450
An impact score of 80 at the 59th percentile means either the study section discussed all proposals, or someone took an interest in your otherwise triage-destined application and advocated for its discussion.
The new scoring range is from 10-90, with impact scores under 20 most likely to be funded … perhaps under 25 since ARRA has relieved the system of some of the backlog of potentially fundable applications. Percentiles issued in this round reflect only the current study section in which the application was reviewed (assuming the study section had 25 or more applications under review … if fewer than 25, then pooled with either all CSR or all IC applications, as appropriate to the review group’s affiliation).
You can find details on the scoring procedure and interpretation as well as reviewer guidelines by mechanism at the Enhancing Peer Review Website.
I’m guessing NIAID will be the first to update their paylines accordingly, and I’ll post updates as the ICs provide them here in the main blog and on the NIH Paylines & Resources page.
Medical Writing, Editing & Grantsmanship 
iGrrrl said
Some of the feedback I’m hearing from those on study section last month is that reviewers were mostly following NIH’s request that they use the entire ranking, with 1 being the exceptional score, and 2 being excellent and fundable. People were giving very, very few 1s, but 3s and 4s quite easily, as well as down to 7 and 8, so I suspect that even scores over 20 could be fundable.
We’ll see…
D said
I bet that this won’t last long though. When the percentiles drop to 5 and the reviewers get used to the scoring system (that is figure out what a fundable score is) 50% of apps will get bunched between 15 and 20.
Of course since reviewers, according to TS, are spending 12 hours/app maybe the score spread will stay…….the true audacity of hope.
iGrrrl said
I have info now from four different study sections, and I’m still hearing that the 1 was not used at all. Of course, all the grants that were fundable bunched around 2 and 3, but scores of 4 and below were given to discussed grants.
D said
I have seen a lot more 1s than I would have guessed. I might be focusing on a different group of Study Sections though. Maybe the infectious disease folk think more highly of themselves than the neuro-hackers.
Assuming my quick read of your internet-trail is correct.
BikeMonkey said
Leaving the 1 for once in a lifetime excellence was promulgated by Scarpa in his presentations to study section chairs. This is not using the full range this is truncating tv range. Directly opposite to the supposed decision making literature that nformed this change and opposite to prior sustained drumbeat to use the full range. Unbelievable……
iGrrrl said
My guess is that this was pushed so hard because of the serious culture shift they seem to want. Going from 41 possible scores to probably 9 isn’t easy for reviewers. They can’t give a 1.5; they have to pick 1 or 2. Or 3.
NIH has acknowledged that the new system will lead to more tie scores. This means that program will have a bit more say in funding decisions. Between that fact, the shortened applications, and the changes in the biosketch, I think that NIH has been looking very hard at how NSF does things. And NSF only has five possible scores…
D said
The confusion that I have with the “Party Line” (the new scoring system will lead to more ties giving Program more say in what gets funded) is that they already had that with scores bunched up. These were essentially all ties and yet Program wasn’t happy picking from the group. Why would changing the scoring system to get more ties make their job any easier?
And what secret knowledge or gnosis do the POs have that make them better able to make these decisions? Do they really have a broader or different perspective than standing members of study sections? Sure they can fix the rare egregious wrong but that is independent of the scoring system.
Let’s be honest. These change will not mean much in the long run. The reviewers are smart enough to figure the new system out and, unless NIH gets a lot more money, we will be in the same fix in 3 years.
All that said I have always liked perturbing stable systems to see what happens. Think anthills.
whimple said
So, how are they going to percentile these new scores? Are they going to wait for a couple of rounds of review to see how it shakes out, or are they going to try to percentile grants using the new scoring procedure with grants using the old scoring procedure?
D said
http://grants.nih.gov/grants/guide/notice-files/not-od-09-024.html
Percentile rankings will be calculated anew, starting with scores from the May 2009 cycle of review, and reported to the nearest whole number.
So, percentiles will be delayed until all of the study sections have met. The first round of the new system will only percentile against that round. The second round against the first and the second. Etc.
D said
Be prepared for the correlation between score and percentile to vary quite a bit for the next few rounds.
whimple said
One perhaps unintended feature of the new scoring system we’re hearing here is that study section members still need to justify scoring outside the range of scores assigned preliminarily by the three main reviewers. This “scoring outside the box” usually doesn’t happen. However, with the scores much more granular now, it often occurs that the three main reviewers will all score the grant the same, say ‘2’. This means that the range is ‘2’. To avoid going outside the range, everyone then scores the grant ‘2’ for a final score of 20. This means there will be a pile-up of grants at scores that are integer multiples of 10. You get a distribution of grants with scores from 11 through 19, then a pile of grants at 20, then a distribution of grants with scores from 21 through 29, then another pile up of grants that all get 30, etc.
D said
I have always felt that the concern about reviewers scoring outside the box and having an undue influence (not based on the discussion) on the final score to be overblown.
A good SRO can look at the scores, see the outliers and judge for themselves. In fact SROs could remove the score from the final average.
By forcing reviewers to declare that they are scoring extra high or low basically pushes everybody towards the mean of the assigned reviewers and prevents the spreading of scores that NIH claims to want. When you ask for consensus you get clumped scores.
Of course this is all because of past protests and complaints by belligerent, unfunded applicants so to paraphrase the Bible….you reap what you sow.
BikeMonkey said
I don’t think SROs can remove scores. POs might be able to take this into account, however. I have seen changes in the out-of-range issue about every other round it seems. Mostly about when and how you declare your intent to score outside. Who knew this was such a problem?
D said
Technically, SROs can remove or change scores. That is, the ERA commons site allow it. Whether they do or not is a matter of CSR policy but I think that they rarely do. I agree with your comment about POs. They can make a good argument for funding a grant out of order if it was sunk by 1 bad score.
The out-of-range issue is a big deal because members of peer review panels talk to their colleagues…”your grant was sunk by 1 reviewer who hates you”…If the SRO can say that the issue was discussed and the panel agreed with the 1 reviewer then no appeal.
No SRO likes successful appeals. This is a classic CYA technique.
Neuropop said
So how am I to interpret this set of scores? Primary: 2’s across the board, secondary, 3’s and 4’s across the board and the third reviewer goes 5+! If this is not out of range, then I don’t know what is….And reviewers didn’t even bother to use up their 1/4 page per criterion in their critiques. Boy, do I long for the days of yore.
writedit said
Interesting. I’ve already been sent a summary statement with a huge range of review criteria scores (2-7, but mostly 4s from both reviewers – the reader gave no scores), an 80 as the impact score, and pages (!) of detailed comments on the Approach especially. In fact, all the reviews read like the usual canned material, including detailed dissection of the experimental methods and why they would or wouldn’t work (&, of course, why not). At least no one provided literature citations for the PI to consider. – writedit
bikemonkey said
Ignore the criteria scores, what was the overall impact score range. If they were 4s and the final score was an 80 than it got talked (way) down during discussion. Someone identified a big problemo. Trouble is that this may not be made readily apparent if the reviewers didn’t bother to edit their reviews after the meeting. Gotta talk to the PO in a case like this and find out what happened.
Not to worry. All this advice and more has already been passed along to said PI, who is surprisingly undaunted. I do wish CSR would add a standard explanatory line to summary statements indicating the impact score is not calculated from the review criteria scores. The PIs should know this from studying the Enhancing Peer Review guidance documents if they aren’t reviewers, but, of course, they don’t all keep up with such things. – writedit
bikemonkey said
Neuropop, the “range” is that set by the assigned reviewers. So if you are looking at scores from one of the critiques, by definition it is within the range.
D said
Also,
1). The new scoring system is new to everyone and there is likely to be a lot of variation. Especially with the expedited RC1 apps.
2). If this is an RC1 application some reviewers only had a week or two to read your app and write a critique. So, not much time for in depth critiques.
3). Since you would never see the score sheets without inside information you would never know if you got an out of range score.
writedit said
On #2, I know RC1 reviewers who just received applications and still haven’t submitted their reviews! (first-round reviewers, not editorial board members) I agree that most RC1s will come back with limited commentary, though, and no doubt many with wildly divergent scores. – writedit
D said
One more point. Keep in mind that the true purpose of the subscores is to give some scoring feedback to folks whose apps are not discussed at the review meeting…triaged in the old vernacular. If you got an Impact score the subscores are less important. Although they should give you some guidance on the weaker areas of your application. But the critiques should do that too.
Again, I think that everything is so new that it will take awhile for the scores to have some real meaning. Allowing you to read between the lines.
Neuropop said
Fair enough. This was not an RC1 but a new investigator R01. Hence the rapid turnaround from a diligent SRO. I shall be on the horn to the PO soon enough. I did get an overall impact score and indeed it seems to have been talked way down compared to what the criterion scores seem to indicate. I realize that reviewers are asked to ensure that the overall impact scores are NOT an average of the individual criterion scores. That said, given the paucity of material in the critiques, I am off to Madame Zelda to help me read the tea leaves. I might be able to expense that somehow
D said
Good luck with the tea leaves. CSR SROs are “encouraged” to get new investigator summary statements out in five working days after the meeting. I am not sure why the critiques are so light for a standing committee. Perhaps everyone was busy writing and reviewing RC1 apps.
Let your PO know that you are disappointed with the minimal critiques.
bikemonkey said
Honestly Neuropop, don’t pay any attention to those criterion scores. nobody knows wtf to do with those, nor what they are supposed to mean. I mean seriously. An Investigator who is a 5 versus a 6? Whut? Either they are good, bad or good and need an extra person for specific expertise….
Ashi said
What kind of percentiles are people receiving for different priority scores? I wonder what kind of drop off there is in percentile with a difference in only 1 point.
M said
score = 20, percentile =11
John M said
For what IC does 20 = 11?
D said
Although most ICs are concentrated in a few study sections, it is study section specific not IC specific.
a p said
score = 30; percentile = 13.
John M said
25 = 13%
J C said
Scores and percentages vary. I know one person with a score of 28 = 11%, I have a score of 25 = 13%, and apparently 30 also = 13%. It varies with each study section, so I really don’t know how funding decisions will be made.
D said
My guess is that since the budgets of most institutes are the same as last year the payline this year should be roughly similar if not higher due to ARRA stimulus funds (which probably soaked up some grants that would have been funded with base line money this year). Also, (ignoring the ARRA challenge grant onslaught) I don’t think CSR saw a rise in R01 applications this year. Next fiscal year is a different story.
So, if you institute funded to the 13% last year you probably have a great shot. Good luck.
writedit said
So I was privy to a little assessment sent to BICO of how the new scores are shaping up in one IC at least. I’ll quote most verbatim, but of great interest was the indication that less than a third of applications examined had scores and percentiles. So here it is:
bikemonkey said
“frustration” is an interesting way to put it…
charles said
I got a 60 score on a Challenge Grant, with an 11%!
So I know the funding is out, but the correlation between score and % is an interesting one!
sahsaj said
Is there a specific website to look up RC1 study section make up? I could not find anything on CSR website. Thanks!
D said
Eventually they will be here (http://www.csr.nih.gov/Roster_proto/allother_sep.asp) Click on the study section name in the left column. CSR does not make the rosters available until 30 days before the meeting. So, most of the ARRA SEPs will be available next week. Assuming they haven’t changed the policy for these reviews of course.
If you dig around (start here http://era.nih.gov/roster/) you can find the rosters for all upcoming study sections, including the ones run by the Institutes.
BikeMonkey said
Looks like a ton of the competing supplements were done with SEPs run after standing sections, with the same people…
sahsaj said
Thanks, very helpful, amazing number of mail-in reviewers..It will be interesting to see how the grant reviewing dynamics work.
D said
We received a priority score of 25 for a submission on an RFA, no percentile score was given. Would anyone like to comment on your thoughts on whether this will likely be funded. Thanks.
D (the other one) said
No idea. RFAs are not percentiled. It depends, mostly, on how many apps the IC can fund and where you ranked in the meeting. The only good source of info would be the Program Officer listed on the announcement. You could try the SRO but they “should” just refer you to the PO.
You could always try chatting up a friend from the SEP. š
J C said
Does anybody have any idea what the paylines for F31 grants will be this year? It is hard to find information on the cutoffs for the new scoring system. Old cutoffs often give scores like 140, but I don’t know how that relates to the new scoring system. Does anybody have any percentile information?
D (the other one) said
Fs, Ts and Ks are not usually percentiled at least at most ICs. Unfortunately, you might have to wait until till closer to September (end of the fiscal year) to find out. If I am reading the calendar right they will have to have paylines for both the old scoring system (Oct/Nov last year which for F31s at NIAID is 145) and the new scoring system. That will be confusing.
As an aside AECOM has a nice summary website of R01 success rates with links IC paylines. http://www.aecom.yu.edu/ogs/NIHInfo/paylines.htm
MM said
FY-2009 NIAID payline for new investigators was 25%, which I heard was aided by the recovery funds.
Has anyone heard or is there any speculation if FY-2010 payline may reach that high or will it be back down to 14% as it was in FY-2008.
The reason I ask is that I am sitting on a 19% with an impact score of 30 and trying to figure if I have a chance at getting funded. Thanks
D said
It is still a mystery. Especially since NIH does not have a budget for FY10 yet. Although they have a pretty good guess. Another difficulty is that NIAID (and each of the other ICs) has a quota (an absolute number not a relative number) of new investigators (actually early stage investigators, ESIs, a subgroup of NIs) that they have to fund. They went to 25% to fill that quota. The tricky bit is that they won’t know how close they are to filling the quota until after the second cycle (Oct./Nov. submission). So, my guess is that they will set a low percentile (say 10% regular 14% ESI) at the beginning of the year and revisit later.
What would your PO tell you to do? Resubmit, if you can. If you think you can improve it enough for the next round go for it. Otherwise wait until Jan./Feb. Remember though you only get 1 resubmission.
If you resubmit and NIAID decides they can fund you with the first score they will have you withdraw the resubmission.
If your A1 doesn’t score well enough to get funded start chatting up your PO about the possibility of Special Pay AKA fund it out of order. NIAID Council loves a good ESI closing-down-the-lab-and-leaving-science story.
PS This stuff is confusing.
PPS No one in Program will make you any promises. But, they should give you pretty much the same story as above.
PPPS a 19% for a first submission (my assumption that this was an A0) of an ESI is a pretty good score.
PPPPS this assumes that the 20,000 ARRA grants that aren’t funded don’t all come back next year.
PPPPPS etc., etc., etc.
D said
A scary update from NIAID
http://www.niaid.nih.gov/ncn/newsletters/2009/0701.htm#n02
I forgot that starting with the Jan./Feb. receipt date R01 apps go to 12 pages.
“An even greater unknown will arrive starting with the January 25 receipt date (May for AIDS applications) when shorter research plans will force investigators to leave out or summarize much of the detail they had been including.”
So you might want to resubmit sooner rather than later.
No one knows how this will affect peer review.
Emaderton3 said
Hi,
My K25 award was just reviewed last week (NCI). I received a priority score of 30. Any clue as to how good a score this is for a K award? I’m guessing it is on the edge of the cutoff, but it really depends what the other K awards were like and is probably different than a group of R01s.
family doc said
Boy this is confusing and frustrating. I just got an Impact score on my K23 (NIMH) of 16. But it’s the weekend, so I won’t know until next week what that means and from some of the posts, maybe not until September?
writedit said
Nothing is guaranteed, but a 16 on a K23 should be fundable.
family doc said
So my Council met to review scores but all my PO can tell me now is that “things went well” and they should have funding decisions before November. He also, at one point, stated that a 16 was about equivalent to a 150. By the formula put forth by Emaderton3, it would be equivalent to a 130. So we may be estimating on the low side.
writedit said
Well, for a PO to admit that “things went well” and to not immediately start hedging about resubmitting with the hope that your A0 application would picked up in the meantime … I think you’re in good shape. I suspect the shift up in the old scoring scale reflects the reality of tighter paylines as much as anything else … though perhaps there were that many better scored applications. I’d be surprised in the K23 category, though.
family doc said
Thanks for the encouragement. What’s an A0?
writedit said
A0 = initial submission, A1 = amended (revised) submission … no more A2s, which were the second amended submission
I’m assuming you got the 16 on the first submission of your K23 application, which is really outstanding … no, wait, exceptional. š
family doc said
Ah, I wish. 183 on first submission. This is 2nd submission. Fortunately this application still falls in the 3-submissions-allowed category based on when I first applied, so I could reapply, I think. My PO asked if I’d turned in my JIT documents, but I thought that was something you didn’t do until after funding decisions? (this is my first experience with NIH).
writedit said
Get the JIT info in ASAP if you haven’t already! A direct JIT request – and a PO asking about this – is confirmation that NIMH wants to fund your application. Council wants to be sure there are no outstanding administrative issues (IRB approval etc.) as evidenced by the receipt of JIT material prior to recommending an application for award.
You’re lucky to have a good PO keeping an eye out for you like this and being so responsive and encouraging.
family doc said
writeedit – Thank you for your support and emails. Just for f/u, they sent me an official JIT request in late November and I did end up getting funded at the very end of December.
monique said
I got a score of 20 on an R21.
Any idea what my chances are?
thanks,
m
writedit said
Again, no guarantees, but I would say your chances for funding are very good.
D said
Percentile?
Davis said
Any thoughts on a K01 submission (NIAID) with an impact score of 29?
Thanks.
Redfield said
I also got a score of 20 on my R21. I spoke to my PO thinking that this was going to be a shoe-in. He agreed it was a good score but he was very non-committal. He mentioned that they may do percentiles for R21s and decided on that basis as opposed to a fixed score as it used to be in the old system. Has anyone else heard about this?
D said
Great info. I am sure Program is just as confused about what these new scores mean as anyone. They’ll have to wait and see what the Budget and Grants Management folks say. (That is how many they can fund in total) They probably want to percentile because of the wide variation in absolute scores between study sections. A 20 could be the best scoring great app in one study section and number 10 in another. It would not be fair to pick a score cut off in that case.
N said
Interesting thread. I have a percentile of 17% and a impact of 66 on a “starred” RC1. I am assuming this means that there was a large number submitted for this RFA, and though my impact was low, there was a larger number that was the same or lower. Any other interpretation I should be seeing? I have not received reviewer comments yet.
This process was not enjoyable.
D said
Your interpretation sounds correct. If you assume that 80% were not discussed then a 17% is near the high end of those that were (for clarity 1%-20% were discussed 21% and up were not discussed). This agrees with the 66.
Sorry..
JH said
My F31 was just reviewed by an SRG in mid-July. I got a 29 but can’t tell how near the payline I might be (My institute doesn’t do percentiles).
Anyone have other scores + percentiles to share (specifically from that meeting)? I heard varying things about the frequency of 1s for fellowships v. R## grants (hence my asking).
Thanks much.
J C said
It’s really hard to compare scores from different SRGs even within the same institute. If your institute doesn’t give percentiles then the scores and percentiles from other ones might not be very helpful to you, but it might give you some sort of reference point. I mentioned these scores in an earlier comment, but didn’t say they were for F31s: I got a 25 on mine (13%) with NINDS as my primary institute. My friend got a score of 28 with an 11% in a different SRG at NINDS. My program officer said that there is a good zone, a bad zone and a gray zone. The gray zone is normally 15-20%, and usually either side of that is easier to predict, although nothing is set in stone.
I hope that information helps somewhat. Good luck!
DP said
Similar to N, I just got an impact score of 65 and a percentile of 15 for an RC1. They must have triaged a ton of challenge apps, then used the full range of scores for the small number actually discussed.
writedit said
That’s exactly what they did (& they received over 20K applications, so yes, “a ton”).
You can monitor comments at the thread on RC1 scores (& remaining review procedures) to see how others are faring across the country.
tuck said
We received a 25 on an sbir and was told it was equivalent to approximately a 150 in the old system from someone inside NIH. Anyone else heard any equivalency type scores.
Emaderton3 said
A friend has told me that some are using the following formula to convert the scores:
Old=5*(New-10)+100
This does capture the entire range of scores, but many at NIH are cautioning that you cannot simply convert scores because grants are being evaluated by different criteria. However, I have seen many posts, including here, where people have heard or experienced others converting the scores anyway.
D said
At first, I thought I would hate the formula. But, after trying it out I kind of like it. It does a pretty good job (given all of the standard caveats) or translating the old and new scoring system. I am sure CSR and OER hates the idea of this formula though.
Dookie said
Not sure if this formula quite works. A 30, which is in excellent in the new scoring system, should be considerably better than 200 in the old system. Moreover, my understanding is that very few scores below 20 are given out (I know of a score of 19 being a 1 percentile).
Just based on scores and percentiles above and percentiles I know of for R01s:
19 – 1%
20 – 11%
25 – 13%
28 – 13%
28 – 11%
30 – 13%
31 – 19%
If we use 25-35 being excellent in the new system compared with 150-200 being excellent in the old system, a better conversion would be:
Old = 3.5*(New-10)+100
D said
The RC1s are a bad comparison. For regular Standing Cmtes 40%-50% of apps are unscored. For the RC1s it is closer to 80%-85%. So you need to multiple the RC1 percentiles by at least 2 to get a reasonable percentile/score comparison.
Also keep in mind that reviewers knew what a fundable score was under the old system and would (even after the protestations of the SRO) use that info to decide on a score in the old system.
Next, there was apparently a wide variation in how committees implemented the new scoring system. Some committees gave lots of 1s and 2s. Some mostly 2s, 3s and 4s. The percentiles will even this out.
I agree that a reasonable guide is the adjectival descriptors. But, until we get through a few rounds of standing cmtes the significance of a score will be difficult to judge. The RC1s are just creating more confusion.
RK said
I just got a score of 22 on my K99 and based on the formula the conversion is 160; however, this was a resubmission and the previous score was 165. So after addressing the reviewers comments which were essentially that it was too broad I eliminated some specific aims. However, if the score only changed by 5 points, then what was the purpose of resubmission? Thus, if the new system lead to the same score as the previous one, I fail to see the utility of the new system.
D said
First, the formula is a rough guide. You can not use it as you have,
Second, there is no guarantee that a resubmission of any kind of grant application will get a better score under either score system. I know lots of stories of PIs resubmitting a scored app and getting a Not Discussed. Or, applicants removing an Aim and then told they should have included it. In general if you do what the reviewers say you will get a better score although not necessarily a fundable one. What you really need to know, and no one will (or should) tell you, is how your rank amongst all K99 apps for your IC. If you got the best score of all K99 submitted congratulations!
In general it is how you rank amongst all similar apps that determines your funding.
So, you need to bug your PO to find out when your IC will determine the funding cut off and whether you have a reasonable chance.
RK said
Thank you very much D said.
AC said
What are the chances of score 60 on RO1 against a PA, under new review criteria? Anyone?
Thnx.
AC
writedit said
Sorry to say, really no chance at all of funding. This would be about a 350 in the old scoring system, which usually wouldn’t even be discussed. I’d suggest you use the individual criteria scores and resume & summary of discussion to assess if the problems are fixable and ultimately fundable.
AC said
Can applicants talk to their assigned study section SRA about the score and percentile calculations received from a special panel? What can and can not be discussed with SRA about the review, before receiving the summary statement? Just curious, as I`m a new investigator and dont know the intricacies.
Thnx.
AC
BikeMonkey said
call your PO, not the SRO, AC.
C said
This whole formula business is rather silly. I would not advise using any formula to map the new score onto the old scoring range. Come on, people! Aren’t we scientists?
Amanda Guth said
Interesting thread, but it makes me nervous now. I just submitted my last chance at a K01 and got an impact score of 25. My previous submissions were a 235 on the first and a 175 on the second. I contacted my PO and all I got from him was the following statement:
“Yes, we changed the scoring system and we have not make any funding decisions yet. I will tell you that your score is excellent!. I will contact you later, when you get the summary statement and reach a funding decision.”
It is frustrating that they are so vague as I would like to know if I have a shot or not or whether I need to start applying for funding elsewhere, since this was my last submission!
DontWonnaBeScientist said
That must be a standard answer as I got exactly the same statement – by word! Got a score of 15 and nobody can tell me what it means – going nuts!
DontWonnaBeScientist said
Has anybody heard back from R21’s yet? Just got told by my PO that they probably won’t make any decisions until Nov in order to process RC’s and R01’s first (NIDDK). Any idea what impact score of 15 could mean in that category? Will postponing the decision into Nov affect the funding likelhood from a fiscal year persepctive?
writedit said
I think an impact score of 15 on an NIDDK R21 is pretty secure … if it is an R21 that falls under NIDDK’s preferred use of the mechanism. I have heard they did not fund within-payline R21s that program felt should be R01s. NIDDK has strict – perhaps the most strict – criteria for funding R21s. If you submitted to one of their R21 PARs, you should be fine, especially if your PO did not mention anything about your application being outside NIDDK guidelines for this mechanism. And actually, your PO sounds quite encouraging, which is telling, so I think you can be reasonably assured of funding. Did you receive a direct JIT request?
The delay in funding will occur everywhere at the NIH. After Sept 30, all the IC grants management offices will scramble to close out the FY09 books (FY10 starts Oct 1), and that usually takes much of Oct – esp this year, with all the ARRA funding that had to be spent by Sept 30. Plus, the ICs must deal with delays in getting the federal budget passed, figuring out paylines for the new scoring system (with limited percentile data esp), and figuring out how big a spike in applications they can anticipate for FY10 (due to recycled RC1s, RC2s, etc.). So, Nov is a reasonable time frame for funding notices for grants reviewed this summer and recommended by Council in Sept.
DontWonnaBeScientist said
UPDATE: MY PO told me that no R21s will be reviewed before January (NIDDK). They don’t even have the payline for R01s figured out yet. Frustrating X-mas!
J C said
Should it be taken as a bad sign if there has not been a request for JIT materials yet? My labmate and I (for F31 grants) both received emails a few weeks ago saying that the NIH anticipates funding out grants, however they are “making no obligations at this time.” Our council meeting times were on the same day, but the next day she received a JIT request and I did not. Does that mean that I most likely will not get funded? I have contacted my PO and have not heard back from him. Any insight??
matfive2 said
I submitted a K01 award to the NIA (2/12/2009), and I got a priority score of 31. I spoke with my PO and he said it was a good score, but that he wasn’t sure it was a fundable score given the new scoring system. I spoke with him once more a few weeks ago and he said that he thought the funding score would be 20-21, but it could be 27-28, he wasn’t sure. Council met on 9/22/2009 and my status changed from “council review pending” to “council review completed”. Is that a good sign? Given his past comments, I have been revising the grant application and am planning to submit on Oct 12th. But I would really like to know what the funding score was so that I can at least assess how much I have to lower my score to get funding. Anybody have any ideas or comments?
Emaderton3 said
I thought the resubmission deadline for a K award was November 12?
writedit said
Yes, the date to submit an amended (revised) application is Nov 12. And yes, matfive2 should plan to resubmit (all the change in status means is that Council met), especially in the absence of a JIT request or further word from the PO. Based on the PO’s earlier comment, I’d say you need to knock at least 10 points off your impact score, and hopefully he is willing to advise you as to whether your planned modifications are in the right direction.
family doc said
matfive2 – my Council just met too, and my eraCommons status changed just like yours, and I had the same question. But my PO said they won’t make final funding decisions until near-November, so I don’t think the status change is significant. I agree with the others that resubmission dates for a K are about a month later than new K submissions–that threw me the first time around. Good luck.
matfive2 said
“Aim for knocking at least 10 points of my impact score” That’s what I was thinking as well. All my mentors and consultants agreed that a 31 was a good score. Looking at the sub-scores from each of the 3 reviewers I think I’ll be able to lower the score 4-5 points without much effort. But I am worried about the additional 5 points that I need to shave off.
As for the Nov 12th deadline for resubmissions, a link would be very helpful. The only information I can find suggests Oct 12th
http://grants.nih.gov/grants/funding/submissionschedule.htm
scroll down to K-series
Emaderton3 said
Scroll down just a bit further and you will see K series listed again for resubmissions. Good luck! I am in the same boat with a K25 score of 30.
matfive2 said
Thanks Emaderton3. The Nov 12th date lowers my stress level substantially. Good luck to you as well! I really want to get the resubmission done for that date because of the new 12 page limit for grants that is being implemented in Jan, 2010. This is my first time trying to get a grant, and I hope it gets funded!
As to your comment above writedit. I have the JIT requestin my eracommons. But I was told by my mentors that the PO would contact me if I needed to submit the JIT information.
Thanks! This is a great post for us novice grant writers!
Emaderton3 said
The JIT request in eRA Commons doesn’t mean anything. I actually came across a NIH web page that discussed how this link is always there. Like someone said above, you want to get a JIT request via email from your PO–that is usually a good sign.
Tishka said
I feel really down about my K01 score of 40. The worst part is when I resubmit next March, I will have to cut it down from 25 pages to just 6! This is my first NIH grant submission, could someone please tell me what JIT is? Thanks!
D said
NIAID has a great description of JIT.
http://www.niaid.nih.gov/ncn/qa/justintime.htm#what
Just-in-time refers to information that we ask you to send us after your application goes through initial peer review and is within the range of possible funding. NIH will need this information but does not require it with your application.
Such as
* Other support. See Prepare Your Other Support Submission in the NIH Grant Cycle: Application to Renewal.
* Institutional review board human subjects certification. If human subjects are involved, provide the Federalwide Assurance number (if changed from the initial submission) and the certification date of institutional review board review and approval. Pending or out-of-date approvals are not acceptable.
* Institutional animal care and use committee, vertebrate animals certification. If research animals are involved, please provide the assurance number (if changed from the initial submission), verification of institutional animal care and use committee (IACUC) approval with date, and any IACUC-imposed changes. Pending or out-of-date approvals are not acceptable.
* Human subjects education certification. For grants involving human subjects, provide certification that each person identified under key personnel involved in human subjects research has completed education in the protection of human subjects. See Other Reporting Requirements, Document Training in Research Conduct, and a Sample Letter to Document Training in the Protection of Human Subjects in All About Grants Tutorials.
Tishka said
You guys are the best! I wish I have known this place before my proposal preparation/submission (1st timer)!
AK said
I have a 10 percentile being new investigator with NCI. I am very nervous with new scoring system. what are my chances? I also submitted the JIT. Your input would help me plan. Thanks
D said
I would guess that a 10% as a regular investigator has a great chance of getting funded! I would start counting my chickens.
Wendy said
I’m a first-timer who just received a score of 30 on an F31. Under the new guidelines that says it was evaluated as “excellent.” But I’m not sure of the chances. My advisors say it’s an excellent score and on the cusp…just wondering if anyone could talk me down re: relaying experience with an F31 being funded.
J C said
I know someone who just got her F31 funded with a score of 28. My status still says “pending administrative review” with a score of 25. I think the percentile is more important than the actual score, but I don’t know exactly how that gets factored in given that the new scoring system hasn’t been in effect for all that long. In an earlier post I relayed what my PO said to me: “there is a good zone, a bad zone and a gray zone. The gray zone is normally 15-20%, and usually either side of that is easier to predict, although nothing is set in stone.” I hope that helps a little, and good luck with your F31!
NS said
Have you guys heard back about your F31? I got an impact score of 40 and my PO told me wait till mid-nov that’s when the decisions would be made. I have not been able to get in touch with him since. My score is in the gray zone but I would still like to hear the final decision.
J C said
Not yet. I keep checking eRA commons just in case. I know somebody who got a “we will not be funding you” email, so at least if you haven’t gotten that then it’s not definitely no. I got a “we anticipate funding your application” email, but that was back in September and I haven’t heard a thing since then. My PO has also recently gone MIA, so it’s probably a really busy time for them. Hopefully we’ll hear something soon! Good luck!
NS said
Thanks JC. I never got any e-mails or status updates since July, so it is kind of frustrating. I just got a reply back from my PO saying that the final funding decisions still haven’t been made–check back in mid Jan!!
NS said
So I just heard back from NIH that I have been awarded the F31. But there is a question my adviser asked.
We are not allowed to have support from two federal agencies at the same time. Can we terminate the F31 half-way through if we get another federal grant and if we want to take it?
The question I had was, it seems my project is moving faster and I think I would be done in a year and i had requested two years of funding. Is it a binding contract where I have to be on the support for two years if I can finish in one year with some modifications to my original proposal?
One More... said
Just got my R21 score back under the new system. It is 34. No percentile rank has been released. It is a PAR application with set-aside $. I will resubmit in all likelihood but wanted to chime in.
The RC1 cycle was a bloody mess. My challenge grant got a percentile of 17 but I do not know of anyone that got anything out of that initiative. I felt like it was a complete waste of time.
Good luck!
Jeff said
I just got my scores back for my F32 application. It was my first submission (first grant submission ever actually) and I received a 32. I’m guessing by reading all of this that this score is probably not going to get funded? There is (of course) no percentile score.
writedit said
First, congrats on that great score for your very first grant submission!! As to your question, it depends on a lot of things, especially the IC (and, as Tishka notes, the mechanism). In FY09, the NIAID payline for F32s was a priority score of 136, while NIAMS set theirs at 180. If you can get the program officer to give you an idea of your relative rank (and you can check historic data to see how many awards they make each year), you can get a reasonable guestimate of funding likelihood.
I’d suggest you also post over at NIH Paylines & Resources, where a lot of folks toss up their scores for feedback on where they might stand in the pile of applications.
Tishka said
Based on the score alone, I think it is impossible to know the funding probability. My project office told me that my score of 40 is in the gray zone, but for another type of award/study session the 40 may be a non-fundable score.
Tishka said
I have heard a rumor that almost half of the proposal got “killed” and no further review/score was given to them, even for the K-series awards. Could someone please verify for me if it is true? If it is indeed true, then if the statistics says that 40% of the total number of proposals get funded in 2008, then does it imply that my proposal has a decent chance of being funded at the resubmission? Thanks!
writedit said
This is not a rumor – this is stated NIH grant review policy. It’s called streamlining or triage, the idea being, with far more applications submitted than can possibly be discussed in detail, to eliminate the bottom half of the grants from discussion so as to invest study section time in those most likely to be competitive for funding. Now the SROs also rank order the applications by preliminary impact score to determine the order in which they are discussed (starting with the best preliminary score). Any applications that are not discussed (due to time limitations or having reached the threshold of ~40-50% of applications under review) do not get an impact score … just the individual criteria scores assigned by reviewers prior to the study section meeting. The goal is to discuss about 40% of applications (this varies by study section & the number of applications under review … so if the applicant pool for a specific mechanism is small, all applications may be discussed, which is why some scores are so high – they normally would have been triaged and not discussed). The success rate for RPGs (research project grants) right now is about 22%, so a lot lower; for F32s, the success rate was 30% in FY08.
DontWonnaBeScientist said
I just heard back on my R21 (NIDDK), first submission with score 15, after I called my PO. I was told that NIH currently does not have a budget and that it is totally unknown when they will start looking at R21. What does that mean? Will it not get funded? Does anybody have some additional information on this? (More than) Desperate to hear!
writedit said
Yes, it is true that the NIH (& most of the federal govt right now) does not have an FY10 budget in place but instead is operating under a continuing resolution (CR), which extends the FY09 funding levels temporarily. They still fund grant applications – they just tend to be more conservative in funding decisions for first cycle submissions (Feb-March) until the budget bill passes Congress and is signed by the President (currently, the HHS appropriations bill is awaiting Conference Committee reconciliation – I think the CR goes until Dec).
Your R21 has been reviewed and scored (congratulations on an exceptional score!). I never like to give folks false optimism, but I think it highly likely you will be funded. If it was reviewed this summer, Council should have considered it by now as well, though notices could be delayed a bit due to the budget hold-up. If your application was just reviewed in October, then Council won’t meet to approve the June-July submission grants for funding until next Feb (earliest start date is in April). You will know you are likely to be funded if you receive an e-mailed (versus standard eRA Commons) JIT (just in time) request.
Unfortunately, since you sound new, the ESI/NI break only occurs with R01 percentiles, not R21 scores or percentiles. However, I think you can be optimistic that your 15 will be fundable at NIDDK. You can also post your score and questions for consideration by a larger group of your peers at the NIH Paylines & Resources page.
nita said
Is a score of 29 good for SBIR grant?
writedit said
Yes, this is a good score – not exceptional, but with a reasonable chance of funding, depending on the IC and how the other applications in the pile were scored. Unfortunately, converting the current scores to the old system (yours would be somewhere between 167-200) is not full-proof since we do not know how reviewers are using the new scoring system. If you can get an idea of your rank order from the program officer and the number of awards they intend to make in your cycle, you’ll know better your likelihood of funding.
Denis said
It’s just sickening right now. We thought that grants would be easier to get since the stimulus – but, the opposite has happened. The success rates are hovering around 6%.
The irony of all is that some senior faculty have up to 5 R01’s, while junior faculty have to scramble for leftovers (R0-Nothings). We’ve got to even the playing field, our the next generation of scientists is going to be forced to give up. We were told it would get better and that there would be a mass-retirement, but that certainly has not happened.
D said
Seems that I have been hearing about “massive faculty retirements” for 25 years now. R0-nothings….I hadn’t heard that one before. Very ironic.
JW said
Can somebody please tell me how the new impact scores are calculated? What I mean is, I got my summary statement back for an F32 application and in each of the critiques there are five scores for different criteria. I had thought that the impact score would be an average of these but that does not seem to be the case. Playing with the numbers it seems that if there was a weighted average the “Research Training Plan” would be weighted at 90+%, which then makes me wonder what the point is of the other sections? Or is the impact score just given by the reviewers independent of their critique scores? Thanks!
writedit said
The Impact scores are not calculated at all and have nothing to do with the scores assigned to the individual review criteria. No weighting, no nothing. You can have review criteria scores all over the map that have no bearing on your impact score, and this can go in either direction (assortment of lower review criteria scores and higher impact score, or low impact scores and several high review criteria scores). NIAID covers the difference between impact and review scoring in their online tutorials.
matfive2 said
This link should provide you with the information you are looking for JW.
http://research.sfsu.edu/blog/post/2009/09/01/new-NIH-scoring-system-Part-Deux.aspx
JW said
Interesting….thank you both for your comments!
K23 said
Can someone tell me how fundable an NIAID K23 with a score of 29 is (my first submission got a 206)? I think I ranked 3 out of 12 in the grants that were actually reviewed. I don’t understand the formula for conversion from old to new. It seems you should just divide in 2. Thanks.
J C said
I just received a Just-In-Time request for my F31 that was submitted in April. I don’t know how quickly things move along after that, but at least something is happening and NIH isn’t at a complete standstill right now…
DontWonnaBeScientist said said
Not sure that this is a proof of moving things forward. I got an automated JIT in August, and have not heard ever since. PO said no results before January (see above comments). With Xmas coming up soon, NIH does seem to be a standstill to me….
J C said
I don’t think mine was an automated comment, though I could be wrong. It was sent from my grant management specialist saying that she was currently reviewing my application and needed three things to complete the review (IACUC, tuition, and documentation that I wasn’t on another NRSA grant). The F31 process sounds like it is very different from RO1’s, but my labmate got her award notice a few days after receiving that email. Either way, I’m hopeful that something is happening and it seems like it’s moving in a positive direction.
writedit said
Agreed, JC, that you got the sort of heads-up that a Notice of Award is likely to arrive soon. Congratulations and good luck with your training and research!
Redfield said
I just read that NIAMS will be funding R21s at the 13 percentile for 2010
http://www.niams.nih.gov/About_Us/Budget/funding_plan_fy2010.asp
I did not receive a percentile with my R21 (through NIAD). Has anyone with an R21 gotten a percentile score? Just curious to see how the priority scores for R21s are translating to percentiles….
D said
NIAID doesn’t usePriority Scores/Percentiles for R21s. They fund by your impact score. Last year, for what it is worth, they funded up to 137. http://www.niaid.nih.gov/ncn/budget/paylines.htm
Redfield said
My PO told me that this year they may start using percentiles for R21s. I guess that’s what NIAMS is using now…
writedit said
By way of reminder for folks discussing scores & paylines here in this post’s comment thread, there is an even longer discussion at NIH Paylines & Resources, and I just started a Discussion page where blog visitors can post their own questions, advice, suggestions, etc. unrelated to a specific topic.
Chris Parsons said
Has anyone heard news of paylines for SBIR/STTR grants yet for FY10? Interested in particular in NIAID and NCI paylines for these-Thanks!
J C said
As of last night, my status on eRA commons changed from “Pending Administrative Review” to “Award prepared: refer questions to Grants Management Specialist.” My project start date changed from 12/1/09 to 07/15/10. Any idea why there is such a long delay? Is this common?
D said
My guess is that they didn’t fund you at September Council (when did your Study Section meet?) but with an actual budget they can fund you after May Council meets. Two months from Council to Start Date is a reasonable amount of time for a GMS to get the paperwork together and send a check to you school.
Assuming I am right..Congratulations.
NS said
D is that true? I got an award e-mail where they asked me what would I like my project start date to be. However, it seems that it takes them long time to process it. My commons doesn’t show any updates since july 2009 when SRG review was completed. Should I e-mail my GMS to find out what the approximate start date would be?
J C said
For mine, it turned out that my notice of award was sent out yesterday, and the July date is the latest I could start the award… so it worked out even better than I was hoping! š
writedit said
Great – congratulations!
D said
NS, In general it is true. But for specific cases it could be faster or slower. Go ahead and contact your GMS and ask. If they give you an option of an award start date then take them up on it. Now, as to what the optimal start date would be….either ASAP (desperately need the money) or delay it as long as possible (give you more time to generate data for you resubmission).
I am a little surprised that commons hasn’t been updated. Was the award E-mail labeled as a Notice of Grant Award (NoA)?
D said
Also, keep in nmind that there are 27 ICs (Institutes and Centers) that make NIH grant awards. Each does things a little differently.
ManD said
Priority score of 24 on an R21 for a special emphasis panel. My “home” institute is NICHD. PO says a percentile will be coming out in the next 2 weeks but can’t give me any sense of what mine will be but that it needs to be less than 13% for fundability. Any clues from others? This was my second of 2 submissions, so there will be no chance to make improvements (my first score was a 60, with a percentile of 52)
writedit said
That’s a really nice improvement in score, but my hunch is this will be close. NICHD has never been forthcoming about paylines, so you will need to rely on the PO for any sense of your ranking and whether your proposal is of sufficient interest and priority to warrant special consideration. Input from others on their NICHD scores/percentiles and funding success would be great, too. Was the SEP for an RFA or PAR or because you serve on the study section?
writedit said
Actually, over on the NIH Discussion page, sr has posted an NICHD R01 with a score of 22 as being in the 14th percentile. Typically, R21s are in the same range … or a bit more competitive (i.e., higher percentile for a given score).
One more said
@ManD
You should be in VERY good shape. I got a 34 and this council meeting for NIAAA had a cut-off at 30 for R21s. Unless it is NIAID or NCCAM, you should get funded.
C D said
I got a priority score of 62 for my K99. Can anyone tell me the possibilities of getting funded.
Thanks
Bad Tidings said
A 62 impact score? It depends on the Institute, but I would say the chances are zero or less.
C D said
It is at NIAMS. Will writing response to critiques comments help? I have not yet got my summary statement.
JD said
Hi everyone:
My K## got a score of 25, and the institution’s funding plan says “payline for K## application will be through priority score of 25” Does that mean mine will be funded? Does an institution ever change (raise) the payline after the fact?
writedit said
“Through” should mean including impact scores of 25. ICs occasionally raise their paylines once a final budget is in and the number of applications to be reviewed is known, but this isn’t anything I would recommend anyone count on. It seems you should be okay with a score of 25, and your PO should be able to confirm this for you. Congratulations!
JZ said
My R01 scored 20 (14%). It is a NIAID’s grant. NIAID’s interim payline for R01 is 8 percentile. Do you have any idea what the final payline would be. Do you think my grant can get funded?
MT said
Damn. I’m now pessimistic on the 32 my P grant renewal application received after reading all the comments here.
Given that part of the reason for the new scoring system was to fight score creep, I assumed that some sort of bell curve would exist with the scores meaning that a 32 should correspond to something like a 15th percentile which would probably be below the payline. Now I have no clue.
ManD said
A follow up – my R21 to NICHD with a priorty score of 24 had a percentile score of 12%. The PO said it would be close, but I was told earlier this week that 12% is “within the fundable range” so I got the grant!
On a related question, does anyone have experience with CDC funded grants? I got a score of 27 on a U grant, but have no clue what paylines are for this agency.
Seungil Ro said
I just got a impact/priority score of 21 for my R21, but no percentile score was given. Can anyone tell me the possibilities of getting funded?
Thanks!
writedit said
That could be a relatively competitive impact score, but it depends on the IC involved. No paylines will be set until next year, though, because the federal government will likely be operating under a continuing resolution until then, and the NIH is facing budget cuts so will be very conservative until funds for FY11 have been appropriated. These sorts of discussions go on mainly at NIH Paylines & Resources (https://writedit.wordpress.com/nih-paylines-resources/), so you could post your score there for feedback … but no one will have much advice to give for several months. In the meantime, keep checking with your program officer about whether your score looks to be fundable or if you should plan to resubmit.
Lee d said
F32 impact score of 30 and 22 percentile, does anyone know if ninds funds f32 based on impact score or percentile?
writedit said
They don’t publish a payline but would likely use percentile for ranking applications. You can ask your PO how your application ranks and whether it is likely to be funded. You might also follow the discussion at https://writedit.wordpress.com/nih-paylines-resources/, where folks share their scores and what they’ve heard from POs.
Gulab said
I just got my K99 score from NEI. Its 21. Does anybody has any idea if there is chance of funding.
Thanks
writedit said
Congratulations on the excellent score … but I am thinking NEI funds down in the 10-12 range since they make so few of these awards (4 per year) … but it depends on how the other applications scored, of course. Your PO should be able to give you some insight into your chances.
gulab said
Thanks. NEI gives 4 grants per year. Its going to be tough to say right now.
SnowballChance? said
Hi, I got a 20 impact score in response to a multi-IC/cross-NIH/OppNet RFA (RFA-HL-12-037). The RFA indicates funds for 2-4 awards are anticipated. On my Commons, NHBLI is shown as having the primary assignment. The application shows JIT now (of course). Thoughts on my chances?
KL said
I have got a 21 (2%th) on an R01 in response to a PAR. I though PAR are usually not percentiled. Is the 2%th real? What is my chance of getting funded in this case.
writedit said
For a PAR, a score of 21 could indeed be in the 2nd percentile, if everyone else has been higher scoring in your study section. I think you can assume you will get funded, but the PO should be the one to say for sure – and should be happy to (since you have a clearly fundable score). In the meantime, you might want to read/follow the main page covering these sorts of questions: https://writedit.wordpress.com/nih-paylines-resources/
KL said
Hi, thank you for your swift response. I was curious because I read from your sites that they are so many people with impact score of 10, 12, 15 in their R01 application, and I got a 21 (and it said 2%ile). Can I assume this 2%ile is close to another 2%ile in the overall pool of scored applications or just this PAR?
Assuming it is not a PAR, what will a score of 21 be like in terms of percentile for an R01 application? Please excuse my ignorance, but why did you think that it is a “clearly fundable score” if there are people with scores like 10 or 12?
Thanks, KL.
writedit said
You are correct, that usually the score is much lower to be in that percentile. You could e-mail the SRO, to be sure there wasn’t an error in calculation. The percentile is specific to the study section, so does not reflect your ranking among all R01s submitted. Because the payline for R01s is generally based on the percentile (except RFAs for which no percentile is assigned), your 2nd percentile should be almost assure funding, assuming there are no administrative or programmatic (i.e., already funding other applications pursuing your science) issues. Your PO can tell you where you stand with regard to funding likelihood.
KL said
Thank you again, I have called my PO, but she is not around. I left a message and am waiting for her to call me back. I will of course keep trying. Meanwhile, I want to thank you again for taking the time to answer my queries. It is great to learn that someone who cares about my situation at this taxing time.
writedit said
Happy to help – hate to see anyone stressing over a simple question when you’re all already stressed by the funding situation and career pressures. Don’t forget to check out NIH Paylines & Resources (and the archives) for lots of shared experiences by investigators in the trenches.
Luck said
I just got my K grant scored 35 from NIAID. Do I have any chance it will be funded?
writedit said
Unfortunately no. The NIAID FY15 K payline will not rise 10 points above the FY14 payline of 25. When you have your summary statement, your PO can help you revise the proposal for resubmission (whether as an A0 or an A1). You might want to follow these pages to see discussions of priority scores and percentiles and which ICs are paying at what level and when (plus other helpful discussions): https://writedit.wordpress.com/nih-paylines-resources/ and https://writedit.wordpress.com/nih-paylines-resources/discussion-nih-scorespaylinespolicypeer-review/
Luck said
Thank you for your prompt reply.