GAO Cautions NIH About Discretionary Funding Decisions

Among other things, in a report entitled “Completion of Comprehensive Risk Management Program Essential to Effective Oversight.”

Why did the GAO do this?

Allegations involving one institute raised questions about areas of oversight by the OD. In light of these questions, GAO examined (1) how NIH makes extramural research funding decisions and OD monitoring of this process, (2) the design of selected internal controls over NIH’s travel and personnel appointment processes, and (3) the design of NIH’s new risk management program and the program it is replacing. To address these objectives, GAO reviewed relevant NIH policies, procedures, and supporting documentation. GAO also selected 3 institutes [NCI, NIDDK, NIAAA] that varied in size for in-depth reviews.

On point one, the GAO recommended that the NIH Director “monitor the extent to which IC directors use discretion in funding decisions,” a suggestion HHS did not embrace.

The recommendation for better oversight derived from the GAO finding of a jump in awards funded above the payline:

In reviewing the IC data, we found that 18.5% of NIH’s funded R01 grant applications were funded as exceptions in fiscal year 2007, as shown in table 1. These applications had scientific merit scores that were below the payline for their respective ICs and thus were funded based on factors in addition to their scientific merit scores. This represents a substantial increase from 9.7% of funded applications that were exceptions in fiscal year 2003.

Of course, the explanation for this jump is exactly what you would expect:

Documentation that we reviewed from three of the ICs—NCI, NIDDK, and NIAAA—showed that IC directors funded applications as exceptions for various reasons. For example, IC officials cited the NIH-wide initiative to fund new investigators as one of the most frequent reasons for making the exceptions. In addition, IC officials told us that they funded applications as exceptions in order to maintain a diverse portfolio of research topics.

Still, the GOA wasn’t happy:

We found gaps in NIH’s ability to monitor key aspects of its extramural funding process. Specifically, NIH’s OD does not monitor extramural funding decisions in which IC directors exercise their discretion to skip applications and make exceptions, even though information on these decisions is collected at the IC level. Without routine monitoring, which is consistent with federal internal control standards, NIH does not have the information to be reasonably assured that these decisions are appropriate and support the agency’s mission. Appropriate funding decisions are critical to ensuring an effective use of taxpayer dollars and supporting NIH’s reputation as the premier federal medical research agency in the United States.

To ensure effective oversight of extramural funding decisions, we recommend that the Director of NIH establish a process for routine monitoring of the extramural funding decisions in which the IC directors use their discretion to skip applications or fund applications as exceptions.

And here is how the GAO characterized the response from HHS:

HHS disagreed with our recommendation that the Director of NIH should establish a process for routine monitoring of the extramural funding decisions in which the IC directors use their discretion to skip applications or fund applications as exceptions. In its written comments, HHS stated that we implied an inappropriate role for the NIH OD. Specifically, HHS said that the OD’s role was not to provide input on the scientific reasoning for making skips and exceptions, which should be left to the judgment of the scientific officials who understand the current trends in science and the institute research portfolios. HHS further stressed that the ICs are required to document the reasons for these decisions and that the documents are available for review by the OD upon request.

In related comments, HHS drew attention to our finding that the share of RO1 grants awarded outside the payline (as exceptions) increased substantially from fiscal year 2003 through fiscal year 2007, and noted that this increase resulted largely from a corresponding increase in the number of RO1 grants awarded to new investigators. We agree with HHS, and noted in our draft report that our analysis of NIH’s records showed that the NIH-wide initiative to fund new investigators was one of the most frequently cited reasons for funding an application as an exception. HHS further stated that it would like to review our methods for quantifying the number of extramural grants funded as exceptions. As we indicated in the scope and methodology section of our draft report, we based our analysis on data provided by NIH. We noted that NIH provided us with information about the payline established by each of the 24 ICs for each fiscal year from 2003 through 2007, and the number of RO1 grant applications funded relative to each IC’s payline for each year.

Embedded in the appended response from HHS is a line graph showing the total number of R01s funded above the payline and the number of such awards made to new investigators. In 2007, 532 of 1,059 raise-to-pay awards went to new investigators. For the other years provided, the data are 203/620 in 2006; 176/572 in 2005; 133/578 in 2004; and 124/625 in 2003. At NCI, which in 2007 had 656 applications scored below the 15th percentile (the payline that year) and 3,108 applications scored above, 3 of the within-payline applications were passed over for funding, while 137 of those outside the payline were funded as exceptions.

12 Comments »

  1. BikeMonkey said

    !!!!!!!

    /pops corn, settles in…..

  2. This is a totally rational response on the part of the ICs to the pernicious combination of the tight paylines and the absurd idea that there exists any actual objective difference in the scientific merit of a 10%ile grant and a 20%ile grant.

  3. BB said

    “…and the absurd idea that there exists any actual objective difference in the scientific merit of a 10%ile grant and a 20%ile grant.”

    Can we tatoo that statement on all who serve on study section?

  4. BikeMonkey said

    The GAO seems to be ignoring the big changes in payline which would seem to be an important consideration.

  5. whimple said

    I wonder what makes the NIH discretionary decisions worse the the discretionary decisions of the study section? Presumably the NIH has interests in the broad health of scientific research generally (such as ESIs) that the study sections do not have.

    • D said

      My reading of the report is not that the GAO cares whether the ICs use skips or exceptions but rather who is watching over the IC Directors to make sure they are making reasonable decisions. That is who is watching the hen house? The farmer or the fox?

      The GAO wants the NIH Director to do this. HHS responds that it is not the NIH Director’s job nor does he have the expertise to do this for all of the institutes.

      I just can’t wait to see if someone publishes the skip/exception lists.

  6. The GAO seems to be ignoring the big changes in payline which would seem to be an important consideration.

    Isn’t that what I just said, cockwad?

  7. mat said

    http://www.nigms.nih.gov/research/application/trends

    very interesting.

  8. writedit said

    Science offers its take on the report, including which IC was at the center of the allegations … and who was doing the alleging:

    Senator Charles Grassley (R–IA), who has also been investigating financial conflicts in medicine, requested the report after questions arose in 2007 about proposals with below-payline scores being funded by NIH’s environmental health institute.

    • Neuro-conservative said

      Now this whole kerfuffle makes a bit more sense. Type “NIEHS” and “scandal” into google for more info.

  9. D said

    After reading this and other similar articles (http://www.nytimes.com/2009/09/22/health/22grant.html) I am still not sure what the problem is.

    I particularly love this quote from the NYTimes article:

    .. Peter Farnham, a spokesman for the American Society for Biochemistry and Molecular Biology, said the health institutes had gone too far.

    “At 19 percent, they’re taking too much from the more seasoned investigators,” Mr. Farnham said. “We would be comfortable with a somewhat lower level.”

    Talk about saying nothing.

    NIH and Zerhouni were very open and clear about finding a way to increase the number of grants going to new investigators. Without more moolah from Congress or cloning and hiring Writedits to go help every new investigator with their grants, the only choice is to “take away grants from more seasoned investigators.”

    The data that I think would be most interesting is the number of these exceptions that scored (old system) outside the outstanding/excellent range. If a lot of very goods/goods got funded then you can make the argument that NIH was over ruling the study section judgment.

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: