Among other things, in a report entitled “Completion of Comprehensive Risk Management Program Essential to Effective Oversight.”
Why did the GAO do this?
Allegations involving one institute raised questions about areas of oversight by the OD. In light of these questions, GAO examined (1) how NIH makes extramural research funding decisions and OD monitoring of this process, (2) the design of selected internal controls over NIH’s travel and personnel appointment processes, and (3) the design of NIH’s new risk management program and the program it is replacing. To address these objectives, GAO reviewed relevant NIH policies, procedures, and supporting documentation. GAO also selected 3 institutes [NCI, NIDDK, NIAAA] that varied in size for in-depth reviews.
On point one, the GAO recommended that the NIH Director “monitor the extent to which IC directors use discretion in funding decisions,” a suggestion HHS did not embrace.
The recommendation for better oversight derived from the GAO finding of a jump in awards funded above the payline:
In reviewing the IC data, we found that 18.5% of NIH’s funded R01 grant applications were funded as exceptions in fiscal year 2007, as shown in table 1. These applications had scientific merit scores that were below the payline for their respective ICs and thus were funded based on factors in addition to their scientific merit scores. This represents a substantial increase from 9.7% of funded applications that were exceptions in fiscal year 2003.
Of course, the explanation for this jump is exactly what you would expect:
Documentation that we reviewed from three of the ICs—NCI, NIDDK, and NIAAA—showed that IC directors funded applications as exceptions for various reasons. For example, IC officials cited the NIH-wide initiative to fund new investigators as one of the most frequent reasons for making the exceptions. In addition, IC officials told us that they funded applications as exceptions in order to maintain a diverse portfolio of research topics.
Still, the GOA wasn’t happy:
We found gaps in NIH’s ability to monitor key aspects of its extramural funding process. Specifically, NIH’s OD does not monitor extramural funding decisions in which IC directors exercise their discretion to skip applications and make exceptions, even though information on these decisions is collected at the IC level. Without routine monitoring, which is consistent with federal internal control standards, NIH does not have the information to be reasonably assured that these decisions are appropriate and support the agency’s mission. Appropriate funding decisions are critical to ensuring an effective use of taxpayer dollars and supporting NIH’s reputation as the premier federal medical research agency in the United States.
To ensure effective oversight of extramural funding decisions, we recommend that the Director of NIH establish a process for routine monitoring of the extramural funding decisions in which the IC directors use their discretion to skip applications or fund applications as exceptions.
And here is how the GAO characterized the response from HHS:
HHS disagreed with our recommendation that the Director of NIH should establish a process for routine monitoring of the extramural funding decisions in which the IC directors use their discretion to skip applications or fund applications as exceptions. In its written comments, HHS stated that we implied an inappropriate role for the NIH OD. Specifically, HHS said that the OD’s role was not to provide input on the scientific reasoning for making skips and exceptions, which should be left to the judgment of the scientific officials who understand the current trends in science and the institute research portfolios. HHS further stressed that the ICs are required to document the reasons for these decisions and that the documents are available for review by the OD upon request.
In related comments, HHS drew attention to our finding that the share of RO1 grants awarded outside the payline (as exceptions) increased substantially from fiscal year 2003 through fiscal year 2007, and noted that this increase resulted largely from a corresponding increase in the number of RO1 grants awarded to new investigators. We agree with HHS, and noted in our draft report that our analysis of NIH’s records showed that the NIH-wide initiative to fund new investigators was one of the most frequently cited reasons for funding an application as an exception. HHS further stated that it would like to review our methods for quantifying the number of extramural grants funded as exceptions. As we indicated in the scope and methodology section of our draft report, we based our analysis on data provided by NIH. We noted that NIH provided us with information about the payline established by each of the 24 ICs for each fiscal year from 2003 through 2007, and the number of RO1 grant applications funded relative to each IC’s payline for each year.
Embedded in the appended response from HHS is a line graph showing the total number of R01s funded above the payline and the number of such awards made to new investigators. In 2007, 532 of 1,059 raise-to-pay awards went to new investigators. For the other years provided, the data are 203/620 in 2006; 176/572 in 2005; 133/578 in 2004; and 124/625 in 2003. At NCI, which in 2007 had 656 applications scored below the 15th percentile (the payline that year) and 3,108 applications scored above, 3 of the within-payline applications were passed over for funding, while 137 of those outside the payline were funded as exceptions.