Medical Writing, Editing & Grantsmanship

My standard advice is always to apply when the application is ready – strong, competitive, and not submitted just to get feedback (especially now with just one resubmission). I always give caveats on the various standard receipt dates and review cycles, such as Cycle 1 carrying the highest risk of delayed or deferred funding. Now, NIAID (who else) has a nice table summarizing all this for me.

Other great articles in this week’s issue of the NIAID Funding News cover the roles and responsibilities of the Advisory Council , whether securing an R21 helps new investigators receive an R01 later (yes, by a 2:1 ratio), and conducting your own “peer review” prior to submitting an application to the NIH.

I took a pass in September on noting Les Costello’s piece in The Scientist entitled NIH R01s: No Longer the Best Science, in which he expresses concern over policies designed to increase funding to new/early stage investigators. This month, Walter Schaffer and Sally Rockey from OER (NIH’s Office of Extramural Research, which brings you the NIH Guide, Extramural Nexus, NIH Regional Grant Seminars, and all you need to know about grant application and management policies) respond with NIH Continues to Support the Best Science through R01s. {see Alison McCook’s comment below for the corrected subtitle to the Schaffer-Rockey piece.}

Essentially, Schaffer and Rockey lay out the history of NIH’s efforts to promote funding to new and early stage investigators (ESI) and the rationale for doing so:

When Dr. Costello [who "vehemently" objects to the new/ESI policy] received his first traditional NIH research grant (R01) in 1963, success rates were near 58%, and 35% of the competing R01s went to first-time recipients. … In 1977, the average age of new investigators was nearly 37, success rates had decreased to 28%, and the proportion of R01s going to new investigators had decreased to 33% … by 2006, less than 24% of the recipients of competing R01s were new investigators, success rates were below 21%, and the average age at first award of an R01 had increased to more than 42.

The first comment, however, keeps on with The Scientist theme about whether the NIH is funding the best science … not necessarily due to any potential discrimination favoring new investigators so much as penalizing amended applications by percentiling them separately (intended to reduce review burden by funding more A0s than A1s based on historical data showing that ~70% of A0s are eventually funded as A1s or A2s). The opening line probably sums up the scientific community’s mood though:

The angst over new and early stage scientists indicates a broader anxiety among established NIH investigators over what is seen as administrative meddling adding to an already capricious peer review process.

And then, of course, we have the forum over at Genome Technology asking Is Peer Review Broken? (review of grant applications and journal manuscripts). Clearly, CSR and OER need to keep the communication channels open for continued feedback on their enhancements to the application and review processes.

So asks the cover story by Meredith Salisbury for Genome Technology.

She starts the ball rolling in her accompanying editorial:

Get three scientists together, and it’s almost a guarantee that the conversation will eventually turn toward the vagaries of the peer review process. Be it for winning grant funding or getting a paper published, this system of relying on a handful of fellow scientists to select the most promising and influential research shapes — at least to some degree — every single researcher’s career path.

And then she gets 3+ scientists together, with the tone set right off the bat by Ferric Fang:

“For something that is of and for scientists, the peer review process is very unscientific,” says Ferric Fang, a professor of laboratory medicine and microbiology at the University of Washington. Whether it’s for papers or grants, having just a handful of people review someone’s work is statistically unsound, he adds. “If these [reviews] were data that you generated in your lab, you would say, ‘I don’t know what the conclusion of this is.’”

And per the suggestions made on grant review processes, apparently efforts to enhance peer review at the NIH haven’t gone far enough. A sampling to get you over to Genome Technology for the full report:

One hope is that having a larger pool of reviewers could help reduce the impact of any individual review, says Fang. Under the current system, “one bad review can sink an application.”

Another take on the grant review system in general is that focus needs to shift away from today’s model of specific proposals for short-term periods. …

Lawrence [Peter Lawrence at the zoology department of the University of Cambridge] would prefer a system where reviewers considered the track record of the investigator more than the details of the new research proposal (with special dispensation for new investigators). …

According to Fang, this concept of awarding funds on a track record basis would also serve the purpose of weeding out people who are very skilled at writing proposals but are less competent at actually performing the science.

Oh no! What’s a writedit to do?! Well, I am the first to acknowledge that no amount of skilled grantsmanship can make up for poor science, so I think, to a certain extent, this last concern can be dispensed with.

The official notice for Restructured Application Forms and Instructions for Submissions for FY2011 Funding is out: all NIH grant applications submitted for due dates on or after January 25, 2010 (i.e., for funding in FY11 and beyond) now have shorter page limits and a restructured format with changes to the research plan, biosketch, resources, and select agent components. The major changes include:

• All Introductions are limited to 1 page (formerly 3 pages for R01s et al.)
• Specific Aims is officially limited to 1 page
• A single Research Strategy (6 or 12 p) replaces Background & Significance, Preliminary Studies/Progress Report, and Research Design & Methods
• Research Strategy organized to address review criteria (Significance, Innovation, Approach)
• Preliminary data/progress reports become part of Approach section of the Research Strategy narrative
• For Select Agents, describe the biocontainment resources available at all performance sites
• Resources must describe environment to support research current proposal (vs only boilerplate)
• ESIs should describe the start-up package/institutional investment in their research career
• Biosketch adds personal statement about why your experience & qualifications make you particularly well-suited for your role
• Biosketch encourages limit of 15 publications

In Science this week, Sudhansu K. Dey, PhD, suggests that the NIH Needs a Makeover to rectify a grant-making process that “has become tangled and inefficient”, mainly due to the lack of quality reviewers. Dey is not speaking as a jilted or jaded applicant, as his impressive NIH funding record demonstrates. Indeed, his concern seems to lie in the fact that his peers (in terms of funding-publications) are not the ones raising their hands to review applications, leaving SROs scrambling to fill study sections:

The Center for Scientific Review is desperate to recruit reviewers and is drafting individuals who have poor records of NIH grant awards or weak publishing histories. How can those individuals be trusted to review grants?

He suggests CSR make more use of videoconferencing to entice participation by busy established investigators, something Toni Scarpa favors as well. However, some reviewers have told me this works better for smaller groups discussing a manageable number of focused applications than for full chartered study sections covering a wide swath of science and funding mechanisms … and that they would prefer not to give up the face-to-face exchange – both to rigorously discuss applications and to network with peers.

Dey also briefly touches on the new scoring and critique system, suggesting he himself is not clear about whether the scores for individual review criteria “have any bearing on the overall score.” Having now seen a variety of summary statements from both discussed and triaged applications for different ICs and mechanisms, I myself wonder – upon reading the bulleted comments in relation to the scores given (criteria & impact) – how well these sync up in terms of the message the reviewers intend to convey. As I noted previously, CSR has a wealth of data to mine in the 20K RC1 reviews, at least in terms of whether the criteria scores match the comments made and send unambiguous feedback to the applicant.

Finally, Dey shares his Makeover advice on the next NIH Director, likely before Francis Collins’ appointment and rapid confirmation:

It is time to appoint a strong leader at NIH who has the understanding of a lifetime researcher and the authority to revolutionize the institution.

Collins is a “lifetime” researcher, perhaps, but one who has spent most of his career within rather than outside the NIH. Indeed, perhaps not surprisingly, Collins does not feel the intramural system is in need of change:

Asked about NIH’s intramural program, he said he is “resistant to the idea that [the program] is in need of some sort of dramatic redo” but is pondering whether to create a pool of intramural money that, like NIH’s Common Fund, could be used to fund crosscutting research quickly.

I suspect Dey was thinking in terms of a “lifetime” researcher who has weathered the turbulence of extramural funding and understands the reality of maintaining a lab dependent on the same. As an aside, Neil Greenspan, MD, PhD, commenting in The Scientist, wonders if Collins is out of touch with reality in a different regard, namely, the realistic potential of genomics in clinical practice. Both of these concerns are dwarfed, however, by the importance of the NIH Director’s ability to get Congress to cough up more money for the base appropriation. Given the current economic climate, Collins has a tough row to hoe here as well.

Last week, a commenter on the RC1 thread asked for reflections on the two-stage peer review process, particularly the Editorial Boards:

Were you on an Editorial Board? I’d still love to hear more feedback on what the reviewers thought of those.

In fact, I’d love to hear more feedback on what applicants and reviewers felt about the Editorial Board review process in general. Independent of the huge numbers, short time frames and 1-2% success rate. Would you want regular study sections to use this process?

Other questions of interest to this individual (and others who have contacted me directly) include:

Was it rigorous? Did it seem like a waste of time? Were the scored apps of high quality? Did you feel rushed? Do you feel that better science got funded by the Editorial Board review process than if grants were picked by random lottery? Did the second level (after mail review) add anything?

Were new/unknown investigators at a disadvantage? Was science outside the interest/expertise of the Editorial Board members at a disadvantage?

I know some questions have come up about conflicts of interest among reviewers, which the NIH recently addressed in its Challenge Grant FAQ:

How were conflicts of interest managed for the Challenge reviews?
Given the volume of applications received and the compressed timeline for finishing the reviews, the NIH determined that it was necessary to recruit over 15,000 outstanding scientists to serve as mail reviewers (including some who would also be applicants). However, a Challenge applicant could only serve in the Challenge reviews as a mail reviewer and not as a study section member, and only for a study section(s) other than the one reviewing his/her application. Mail reviewers do not participate in the discussion or final scoring of the applications, and do not interact with other study section members.

Hmm. Except Editorial Board reviewers were asked to score applications based on the mail reviewer scores and critiques … though apparently most Editorial Board members felt they could not do so without looking at the original application … often leading to critiques of the mail reviewer critiques … and so on.

And heck, why stop at the special process used to review RC1s …. How do reviewers (and, I suppose, applicants) feel about the new review, scoring, and critique procedures?

One Editorial Board member told me that on more than one application, the mail reviewers had very divergent scores but were in agreement with their critiques/opinions, suggesting the learning curve will be steep on the uniform assignment of scores. Perhaps the NIH could use these thousands of clusters of three naive (in terms of the scoring procedure) reviewers looking at the same application to analyze patterns of score assignment against the written comments. I know just the person to write the grant application to fund this …

And what about the plan for increased use of alternatives to in-person study section meetings, which is when many of these finer points would be addressed and, of course, advocates speak out on behalf of specific applications.

Fire away, folks. The NIH needs all the feedback it can get.

I had a conversation some time ago with my Division Chief here at BICO explaining why he could not budget (DC) for someone to handle the ARRA reporting and administrative requirements (special admin costs may become direct when the science itself requires them, such as for large survey research – not when the terms of award do). More recently, the NIH has come out with a clear statement for all those other PIs trying to sneak some ARRA reporting support into their grant personnel costs:

Can ARRA grant budgets include direct costs to address ARRA administrative and reporting requirements?

No. It has been determined that ARRA requirements do not provide sufficient justification to support the provision of direct costs for administrative support in addition to the Facilities and Administrative (F&A) costs in the awarded budget.

NIH staff will continue to evaluate direct costs requested for administrate support using the guidance provided in OMB Circular A-21 F.6.b.(2). However, ARRA requirements will not be considered in this determination, because they are incorporated within the core administrative support for the project, which is reimbursed to the institution through the provision of F&A costs.

When direct costs have been requested to address ARRA administrative and reporting requirements in requested budgets, NIH staff will make appropriate adjustments. Revised budgets submitted solely to remove administrative costs will not be accepted.

Update: CSR has confirmed in the May Peer Review Notes that they will use the 2-phase editorial board approach for the RC1s, as noted in the comment below.

NIH has posted a tentative timeline for the Challenge Grant review process, which will likely follow the Transformative path of a two-stage editorial board review (so assume Stage 1 reviews completed in June, probably via IAR, with editorial boards discussing applications in July):

NIH is using an innovative peer review format—editorial board reviews—to help identify research that has the potential to transform biomedicine. Two groups of reviewers play key roles: specialized experts assess the applications for scientific merit and submit written critiques; experienced reviewers with a broad understanding of the science further critique the applications. This second group functions as the editorial board and meets face-to-face or electronically to discuss the initial assessments and score the applications by focusing on their overall significance and impact.

CSR is using this type of review to assess new Transformative R01 applications in an effort to support extraordinarily innovative, high-risk, original or unconventional research projects that hold promise to revolutionize a broad area of biomedical or behavioral research: http://nihroadmap.nih.gov/T-R01/. Editorial board reviewers will conduct an initial review and select a reasonable number for further review.

CSR initiated an evaluation of the quality and effectiveness of editorial board reviews in spring 2008. The pilot involved a range of SBIR (Small Business Innovation Research) and Bioengineering Research Partnership applications. Preliminary data show that editorial board reviews may improve the quality of reviews for complex, multidisciplinary applications.

A majority of reviewers, some 64%, said they would choose this type of review for their own applications; of the experts who submitted written critiques considered by the editorial boards, about 38% said they would prefer these reviews for their own applications.

No matter what, you’ll receive back critiques and criteria scores (and an impact score if your application is “discussed”), which will be useful for converting RC1s into R21s or R01s as appropriate … or not if the comments and scores suggest you not pursue this line of work.