A2 Nevermore …

Although Nature News Blog reported in October that the NIH was still reconsidering its elimination of A2 applications, today Sally Rockey posted data in support of the sunset policy and ended speculation that a second amended application might again be welcomed at CSR.

In addition to funding a higher proportion of A0 applications (compared with A1), “the average time to award from submission of A0 applications has been reduced from 93 weeks to 56 weeks.”

She also goes on to address the possibility of limited A2 submission for those A1s that scored just outside the payline (whatever that means …):

We looked at this issue using data from fiscal year (FY) 2011. If A1 applications with percentile scores below 25% were allowed to submit an A2 application in FY2011, it would have resulted in 764 unsolicited A2 R01 applications. 165 of the new applications were from new investigators. In addition, only a small minority of eligible applications (37 of 218 renewals) were from investigators trying to renew a previous new investigator award.

Assuming the most extreme case – that all 764 of these A2 applications would have been funded, NIH would have been able to fund 21% fewer A0 applications and 19% fewer A1 applications in FY2011 (figure 3). These displaced A0 and A1 applications would be highly likely to come back as A1s and A2s (as most displaced A1s would become eligible under the modified policy) and the average time to award would increase.

In the end, she notes,

Overall, these data indicate that the policy to sunset A2 applications continues to achieve the stated goals of enabling NIH to fund as much meritorious science as possible in as short a time period as possible. Any revision to the policy to allow additional resubmissions of all or a subset of A2 applications will displace equally meritorious A0 and A1 applications, and increase the time to award for many applications. For these reasons, we have decided to continue the policy in its current form.

About these ads

10 Comments »

  1. whimple said

    They could shave the budgets of the funded A0 and A1 proposals to pay for the funded A2s. I would advise them to allow A2s on proposals that improved significantly from A0 to A1, not the proposals that just missed the A1 proposal payline. A grant that dramatically improves its score means that either the investigator has rapidly improved their grant-writing ability (as might be expected for a new investigator getting feedback for the first time), or the science has rapidly improved (such as with new data, or a newly established experimental system, or a high impact paper). Alternatively, the score may have improved dramatically through random chance (lack of correlation between “meritorious science” and score) which would also be useful information for CSR to have. :)

  2. Jeremy Berg said

    While it is true that the “No A2″ policy is achieving the goal of getting more outstanding applications funded earlier, there are still several points of concern. A major one involves what happens to projects that are not funded after an A1. Often an investigator has many years of effort (and expenditure) associated the project and it may be in no one’s interest to completely change directions to something that the investigator knows (and cares) less about. With success rates where they are now, the science in such projects can be very strong. It may be particularly difficult for early stage investigators to change directions dramatically so that the question of where the bar is set for a “new” application (versus a disallowed A2 application) is crucial. Second, the original “Enhancing Peer Review” working groups actually recommended that all applications should be considered “new” (as a way of dealing with the apparent favoring of applications simply because they were A2s). This approach met with considerable concern from portions of the scientific community and the “No A2″ policy was developed and adopted as an alternative. However, as illustrated above, the “No A2″ policy runs the risk of having strong projects having no real recourse but abandonment or other non-NIH funding sources if the bar for what constitutes a new application is set too high.

  3. However, as illustrated above, the “No A2″ policy runs the risk of having strong projects having no real recourse but abandonment or other non-NIH funding sources if the bar for what constitutes a new application is set too high.

    Jeremy, you seem to be forgetting that arithmetic has still not been repealed. For every “strong project with no recourse” that would get funded as an A2, there is some other strong project that then could not be funded. Allowing A2s *cannot* possibly reduce the number of “strong projects” that go unfunded. All it can possibly do is change *which* strong projects get funded, and there is no evidence whatsoever that it even does change that much, if at all.

  4. whimple said

    Allowing A2s *cannot* possibly reduce the number of “strong projects” that go unfunded.

    Well, not to get all arithmetic on you or anything PP, but you could for example, oh, I don’t know, shave the budgets of the funded A0 and A1 proposals to pay for the funded A2s.

  5. Jeremy Berg said

    Comradde: I understand the arithmetic argument. The concern that I was trying to raise is the following. Suppose a new investigator has invested years and most of their start-up funds establishing a project that does not get funded through an A0 and an A1. If he or she cannot modify the application enough for it to be considered new, then the investigator must find another funding source or drop out of the system entirely, assuming that the investigator does not have the resources available to generate the preliminary data necessary for a competitive application. This is a substantial waste of human and other resources. The alternative would be allowing the investigator to come back with an A0 application even if the science had not changed much. I realize that this will further burden the review system and will not increase the number of grants that are funded, but the investigator could continue to compete for funding. The peer review scoring system is not so robust that an unfunded A1 with a 15th or 20th percentile score is substantially different than an application at the 10th percentile that is funded. Essentially, the same argument applies to an established investigator with a single long-standing project in his or her laboratory. Unless NIH tracks investigators dropping out of the system, they are unlikely to see this potential effect of the “no A2″ policy the way they are analyzing the data.

    The “Enhancing Peer Review” groups were concerned about applications getting better scores just because they were A2s. The “no A2″ policy addresses this but has other consequences. Perhaps these consequences (i.e. “culling the herd”) are necessary given the realities in biomedical research at this point in time, but it would be worth doing the analysis and being explicit about whether or not this is a goal.

  6. Spiny Norman said

    “Often an investigator has many years of effort (and expenditure) associated the project and it may be in no one’s interest to completely change directions to something that the investigator knows (and cares) less about.”

    The First Rule of Holes: if you’re in one, stop digging. The abolition of A2 submissions is useful in part because it can help people to stop digging, and because it prevents the waste of resources (referee time, NIH staff time, PI time) that might otherwise be spent on hopeless holes being dug by overly inflexible PI’s.

  7. Spiny Norman said

    “Perhaps these consequences (i.e. “culling the herd”) are necessary given the realities in biomedical research at this point in time, but it would be worth doing the analysis and being explicit about whether or not this is a goal.”

    This is a key and critical point. NIH, NIH-funded investigators, and other stakeholders would benefit enormously from increased transparency of the sort that Dr. Berg preached & practiced as NIGMS head.

  8. biologist said

    Lets say the payline is 11%. The implication is that an A0 or A1 receiving a 9% is better science and worthy of funding than an A1 that received a 14%. Why doesn’t the NIH simply make this an explicit assertion? Moreover, the implication is that an A0 or an A1 that received the 9% is better science than the A1 that got the 14% even if, as an A2, it would have scored say, 5%! The bottom line is that an A2 making the payline (no matter by what margin) is considered to be worse science, and science not worthy of funding, compared to the A0 or A1 that made the payline. Maybe because having the second chance subtracts from merit? This should be admitted by NIH.

    As far as “culling the herd”, you will never, ever hear anyone at NIH come close to saying this is either an intention or outcome of the policy.

    So, why allow the A1???? Funded A1s displace A0s!! I just had an R21 go from ND on the A0 to 5% on the A1, so I’m not grinding an axe.

    • writedit said

      Not all ICs use a hard payline – each has their own way of selecting applications to fund, and many discuss the science and relevance to mission for all applications within a competitive score/percentile range. The assumption on revised applications is that the PI has made improvements based on the prior critique and new data/reports since the original submission (the merit is based on that application, not in comparison with the prior submission). Given the pace of scientific advance, one would hope that after two years (time scale of A0 & A1 submission/review etc.), the PI would have something new to propose by then. Because reviewer community service is a limited commodity, the goal is to limit burden, particularly when too many applicants submit the same unfundable application as many times as they are allowed just to show their department that they are trying. The alternative to limiting the number of times the same proposal can be submitted would possibly be to allow SRGs to designate more applications as not eligible for resubmission (i.e.,, the reviewers concur the science will never be competitive for funding as proposed due to lack of significance etc.), but that is a huge burden to place on them.

      • biologist said

        @writedit,
        I don’t disagree with anything you say here (except that experientially, I have never seen an IC exercise the kind of discretion you describe. I think that generally they pay very, very few grants out of order, and it is most definitely the exception that proves the rule), but I don’t see how this challenges the validity of my points. I still think the NIH should explicitly say that they think if a grant requires three rounds to get a fundable score, it doesn’t deserve to be funded, but also, why they allow A1s but not A2s. I have a thought or two on this point. One, it could be that the NIH thinks a study section may be wrong once but not twice. Secondly, it may be rooted in a vague sense or belief that everyone “deserves” one appeal on any issue. But, that is wooly.

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 256 other followers

%d bloggers like this: