Reviews of New NIH Review System

Thanks to Jef Akst at The Scientist for his post earlier today entitled NIH Reviewers Praise New Rules (& SaG for the alert). Thanks too to those who in turn have commentend on the piece. I tend to agree with both sets of contributors – the 3 grant reviewers in the main article and the various commenters who have come along since.

Recognizing the learning curve associated with adopting a new system and inherent challenges in assessing significance and impact, reviewers felt that “changes to the reviewing guidelines have actually increased the validity and utility of the reviews.” Specifically:

For example, in the new system scoring is limited to whole numbers (1 through 9), whereas before, a reviewer could give a proposal a priority score anywhere from 1 to 5 in increments of tenths. But such a fine scale was counterproductive, Wiley noted. “[Y]ou cannot possibly discriminate grants on that kind of level,” he said, adding that the process was effectively “a crapshoot after you pick the top 25%.”

Another benefit of the new system is that reviewers are now required to justify their scores by listing strengths and weaknesses for each subcriterion…

Other changes to the review process specifically aim to cut down on the amount of time the process takes. … The template now provided by the NIH gives bullet points where reviewers are to write a couple of sentences summarizing the strengths and weakness of each category, limiting them to just half a page.

Finally, rather than reviewing the grant proposals in random order, the study section starts with the highest scored applications (based on preliminary scores) and works their way down the list. In addition to cutting the total number of grants the study section will review orally — low-ranked applications with no chance of getting funded won’t even be discussed — it also helps the reviewers to “recalibrate” their scores, Wiley said, by providing an excellent standard against which the others can be judged. “It was a very clever idea,” he said. “I found this has been very, very helpful.”

Of course, those fast, easy, bulleted comments are not without their shortcomings, as many commenters noted (e.g.):

The comments are vague and provide little meaningful guideline for improvement. Of course, that is not the primary goal-to be helpful. I think this is mainly a mechanism to more efficiently winnow down the stack of applications in this time of tight paylines.

The new format makes the critique more diffuclt to decipher and the applicant will have a hard time to get a handle on how to revise the application to get a better score.

Plus a reflection on the preliminary score-driven discussion order:

The new review order is truely a clever idea that helps to calibrate the scores and make the whole process more consistent. I delibrately avoid the wording “fairer” because I also see the downside of this new order. The more contentious and contraversial applications are more likely being discussed near the lunch breaks or late in the day and people tend to get really tired by that point. The debate on those contraversial grants may not be as vigorous as it used to be.

I dunno. Looking at summary statements coming out of this brave new world, I’m pretty happy with the way things are going, vague bullets and all. I especially like the ability (or perhaps increased tendency) of reviewers to comment directly on the advisability of resubmission. And I’m happy with the way SROs are preparing the Resume & Summary of Discussion paragraphs as well. I like the short applications. I like the reviews. The paylines, not so much.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: