Medical Writing, Editing & Grantsmanship

In Science this week, Sudhansu K. Dey, PhD, suggests that the NIH Needs a Makeover to rectify a grant-making process that “has become tangled and inefficient”, mainly due to the lack of quality reviewers. Dey is not speaking as a jilted or jaded applicant, as his impressive NIH funding record demonstrates. Indeed, his concern seems to lie in the fact that his peers (in terms of funding-publications) are not the ones raising their hands to review applications, leaving SROs scrambling to fill study sections:

The Center for Scientific Review is desperate to recruit reviewers and is drafting individuals who have poor records of NIH grant awards or weak publishing histories. How can those individuals be trusted to review grants?

He suggests CSR make more use of videoconferencing to entice participation by busy established investigators, something Toni Scarpa favors as well. However, some reviewers have told me this works better for smaller groups discussing a manageable number of focused applications than for full chartered study sections covering a wide swath of science and funding mechanisms … and that they would prefer not to give up the face-to-face exchange – both to rigorously discuss applications and to network with peers.

Dey also briefly touches on the new scoring and critique system, suggesting he himself is not clear about whether the scores for individual review criteria “have any bearing on the overall score.” Having now seen a variety of summary statements from both discussed and triaged applications for different ICs and mechanisms, I myself wonder – upon reading the bulleted comments in relation to the scores given (criteria & impact) – how well these sync up in terms of the message the reviewers intend to convey. As I noted previously, CSR has a wealth of data to mine in the 20K RC1 reviews, at least in terms of whether the criteria scores match the comments made and send unambiguous feedback to the applicant.

Finally, Dey shares his Makeover advice on the next NIH Director, likely before Francis Collins’ appointment and rapid confirmation:

It is time to appoint a strong leader at NIH who has the understanding of a lifetime researcher and the authority to revolutionize the institution.

Collins is a “lifetime” researcher, perhaps, but one who has spent most of his career within rather than outside the NIH. Indeed, perhaps not surprisingly, Collins does not feel the intramural system is in need of change:

Asked about NIH’s intramural program, he said he is “resistant to the idea that [the program] is in need of some sort of dramatic redo” but is pondering whether to create a pool of intramural money that, like NIH’s Common Fund, could be used to fund crosscutting research quickly.

I suspect Dey was thinking in terms of a “lifetime” researcher who has weathered the turbulence of extramural funding and understands the reality of maintaining a lab dependent on the same. As an aside, Neil Greenspan, MD, PhD, commenting in The Scientist, wonders if Collins is out of touch with reality in a different regard, namely, the realistic potential of genomics in clinical practice. Both of these concerns are dwarfed, however, by the importance of the NIH Director’s ability to get Congress to cough up more money for the base appropriation. Given the current economic climate, Collins has a tough row to hoe here as well.

Last week, a commenter on the RC1 thread asked for reflections on the two-stage peer review process, particularly the Editorial Boards:

Were you on an Editorial Board? I’d still love to hear more feedback on what the reviewers thought of those.

In fact, I’d love to hear more feedback on what applicants and reviewers felt about the Editorial Board review process in general. Independent of the huge numbers, short time frames and 1-2% success rate. Would you want regular study sections to use this process?

Other questions of interest to this individual (and others who have contacted me directly) include:

Was it rigorous? Did it seem like a waste of time? Were the scored apps of high quality? Did you feel rushed? Do you feel that better science got funded by the Editorial Board review process than if grants were picked by random lottery? Did the second level (after mail review) add anything?

Were new/unknown investigators at a disadvantage? Was science outside the interest/expertise of the Editorial Board members at a disadvantage?

I know some questions have come up about conflicts of interest among reviewers, which the NIH recently addressed in its Challenge Grant FAQ:

How were conflicts of interest managed for the Challenge reviews?
Given the volume of applications received and the compressed timeline for finishing the reviews, the NIH determined that it was necessary to recruit over 15,000 outstanding scientists to serve as mail reviewers (including some who would also be applicants). However, a Challenge applicant could only serve in the Challenge reviews as a mail reviewer and not as a study section member, and only for a study section(s) other than the one reviewing his/her application. Mail reviewers do not participate in the discussion or final scoring of the applications, and do not interact with other study section members.

Hmm. Except Editorial Board reviewers were asked to score applications based on the mail reviewer scores and critiques … though apparently most Editorial Board members felt they could not do so without looking at the original application … often leading to critiques of the mail reviewer critiques … and so on.

And heck, why stop at the special process used to review RC1s …. How do reviewers (and, I suppose, applicants) feel about the new review, scoring, and critique procedures?

One Editorial Board member told me that on more than one application, the mail reviewers had very divergent scores but were in agreement with their critiques/opinions, suggesting the learning curve will be steep on the uniform assignment of scores. Perhaps the NIH could use these thousands of clusters of three naive (in terms of the scoring procedure) reviewers looking at the same application to analyze patterns of score assignment against the written comments. I know just the person to write the grant application to fund this …

And what about the plan for increased use of alternatives to in-person study section meetings, which is when many of these finer points would be addressed and, of course, advocates speak out on behalf of specific applications.

Fire away, folks. The NIH needs all the feedback it can get.