Medical Writing, Editing & Grantsmanship

Ugh. Nature summarizes the continued (& spreading) negative repercussions of the retracted Dwyer et al. papers discussed here and throughout the scientific community. An excellent Nature feature article by Erika Check Hayden covers the entire saga in detail.

Earlier, our man-on-the-spot David added a nice summary of an article in the May 5, 2008 issue of Chemical & Engineering News, including a bit more insight from John Richard in Buffalo and Hellinga’s dismissive reply (thanks so much, David, for all your contributions).

Nature notes that Richard and his group “wasted seven months and tens of thousands of dollars failing to reproduce the results from Hellinga’s lab” and that Mary Dwyer’s scientific career is over before it started. The editors also observe the questionable rationale in Hellinga’s accusing her of scientific misconduct: “As Dwyer’s adviser, Hellinga was responsible for training her. If she made mistakes, they are ultimately his responsibility.”

And, Nature spreads the wealth by noting that Duke itself owes something to the scientific community, and indeed, one hopes they are not holding their breath and hoping this will blow over so Hellinga can get back to the business at hand: bringing in research dollars to the University.

Earlier discussion can be read first here (initial Science retraction and detailed comments on problems with the science involved) and then here (The Scientist blog coverage, Chemical & Engineering News synopsis, et al.).

The Scientist discusses a case study of a budget-crunch-induced lab closure, the personal and professional fall-out, and the concept of intramural bridge funding. In this case study, the PI (at Man’s Greatest Hospital) received a couple of Type 1 R01s during the NIH budget doubling that were not scored (article refers to them being returned without review - I assume she means without discussion/score) as Type 2s, resulting in funding loss … personnel loss … job changes … the works. Very interesting & engaging series of comments in response as well.

Historically, Type 2 R01s (competing renewals) enjoy roughly twice the success rate of Type 1s: 49.1% vs 24.5% in 2002 … 42.1% vs 20.0% in 2004 … 33.5% vs 16.3% in 2006. Hence the encouragement to pursue R01 vs R21 or R03 funding if possible (since the latter non-renewable mechanisms cannot take advantage of this higher success rate for Type 2s). In 2007, the gap narrowed a tad, with 36.1% success rate for Type 2 vs 19.2% for Type 1. As The Scientist points out, this meant 4,108 Type 2 applicants ended FY07 with out an award in hand.

I was pleased to see The Scientist feature institutional bridge funds. I strongly feel no research institution with a shred of integrity can get by operating without such a mechanism in place and transparent criteria for evaluating requests and distributing funds (distributive & procedural justice). These institutions gladly take the indirect costs their investigators bring in (70% at Dana Farber! and when I left another HMS teaching hospital in the late 80s, it was at … 92.3%!!! compared with a paltry 49% here at Baby It’s Cold Outside). In a cut-throat climate, they also dangle the damnedest recruitment packages to lure/buy funded researchers from other institutions. They better be ready to step up to the plate when the loyal productive PIs need help weathering the storm.

Apparently the Dana Farber will help those with scores within the 15th percentile that still weren’t funded. Baby It’s Cold Outside gives some weight to the percentile but just as much to the summary statement (some scores are outliers due to easily fixable application weaknesses rather than lack of productivity or scientific significance), the PI’s response to the critiques, and other funding sources. Budget requests must be prioritized, with students who need support in that particular lab for that particular project to finish up experiments for their thesis or dissertation receiving the highest priority. Maintaining a specialized animal model/pedigree or ensuring data collection continuity for a clinical cohort rank up there as well. Explicit reviewer requests for specific data to include in the amended application can qualify as bridge worthy. So, some strategies to consider for those of you trying to make a case for internal bridging monies at your home institution.

As discussed here last spring, NIH R56 bridge awards are also available, not via direct application by the affected PI but by recommendation of program directors and ICs. Last January, the Great Zerhouni reissued official notice of the NIH Director’s Bridge Awards. Usually program officers nominate their best & most worthy PIs without saying anything to said PI (so as not to get hopes up, especially when people’s careers are at stake in many cases) … but they shouldn’t mind if you send a quick e-mail reminding them that your A0 was really close and you could sure use a funding safety net until the A1 or A2 is awarded. Now, if your application was more than 10 percentile points above the current payline, probably not much they can do (don’t even think about asking about R56 support for an unscored application).

As I type this, it occurs to me that the case study in The Scientist was unscored but still received $50K a year for 2 years from the MGH as bridge fund grants, so clearly they have more liberal standards than the Dana Farber … or the NIGMS … as well.

Next week, I check out Life on Mars. Maybe it will look better on the red planet.

Update: Thus Spake Zerhouni … the projection of Age Distribution of PIs out to 2020 is amusing.

Update: The Chronicle of Higher Education has an article about NIH efforts to lure back senior reviewers, including an update on pilot peer review reform projects underway and a table at the end showing the percentage of assistant professors serving on CSR panels for 2002, 2005, and 2007. The numbers? 8%, 10%, 7%, respectively.

The Peer Review Advisory Committee (not to be confused with the Working Group of the Advisory Committee to the NIH Director on NIH Peer Review) met today, and the presentations seem to all be uploaded … except for the Great Zerhouni’s update on NIH Peer Review Enhancements. So, we have:

- Toni Scarpa on CSR Initiatives to Improve Peer Review (most bang for the buck re: content)

- Megan Columbus on Electronic Submission Update (transition to Adobe not until Dec 2008 - tentatively)

- Don Scheider on CSR Realignments (neuroscience case studies) & Clustering of Applications (orphan applications explained)

- Marion Mueller on Peer Review in Germany (leads with quote “Peer review is 50% garbage, 50% malice, and 10% good advice.”)

- Olivia Bartlett & Shamala Srinivas on Instantaneous Electronic Scoring of Multicomponent Applications (P01, P50, U19, U54)

Norka Ruiz Bravo has two slides comparing outcomes in women and men and new and established investigators. For 2006, it seems 11% of women and 9% of men scored within the top 10% in CSR study sections. According to the May 2008 council meetings (nice trick to provide these data on April 30th), 13% of new investigators and 18% of established investigators had Type 1 applications scoring within the 20th percentile. Interestingly, the May councils in 2001 and 2002 showed the distribution to be 15% and 17%, respectively, with the gap creeping up in May 2006 (14% vs 18%) and again in May 2007 (14% vs 19%).

In Nature Reviews Molecular Cell Biology, NIAID’s Jonathan Yewdell invites those looking for research career advice in the biomedical sciences to “take a study break, grab a cup of coffee and read on.”

He offers his series “How to succeed in science: a concise guide for young biomedical scientists” in 2 parts.

In the May issue, Part I: taking the plunge, where he provides “unvarnished [not as unvarnished as PP's though] advice for young biomedical scientists on the difficulties that lie ahead and on how to find the right laboratories for training in the skills that you will need to succeed.”

In the June issue, Part II: making discoveries, where he provides “practical advice to young scientists on choosing a research topic, designing, performing and interpreting experiments and, last but not least, on maintaining your sanity in the process.”

Case study from the writedit files with excerpts from a triaged R21 summary statement, which means the PI would have only one page in which to respond to all 3 reviewers. For those of you out there who have not yet had the pleasure of discerning the true meaning of an unscored summary statement, enjoy this taste of what you might have to look forward to.

Reviewer #1: “The application is not competitive, suffers from a lack of precision, and describes a series of weakly related studies aimed at investigating the …”

Reviewer #2: “This excellent application seeks to delineate the interactions of … and their roles in … . The investigator and environment are outstanding, and the work is highly innovative.” [note: "Excellent" refers to score range of 150-200]

Reviewer #3: “As a proof of concept, this application has great appeal. However, it was not clear to this reviewer that … is a worthy target for all of this effort.”

These would be from the “Overall Evaluation” sections at the end of the first two individual critiques (no summary of study section discussion in this case due to traiged status) and extracted from the last of 3 paragraphs provided by the third reviewer. I’m guessing a lot of NIH applicants out there are smiling (grimacing) & nodding knowingly.

Many clues can be found in the summary statement. The second laudatory reviewer (<1 p review) regurgitates the applicant’s main points but provides no real additional commentary. So, not a lot of weight to this “critique” no matter how attractive it might be to consider this to be the only one worth listening to.

The first (2.75 p review) and third (0.75 p review) reviewers cite specific concerns and their rationale for being concerned, which is what adds weight to their input. If the PI were responding, these would be the points to address in the Intro and to correct in the scientific approach & the application.

In its favor, consider that the significance and innovation of the application were uniformly recognized, the PI thought to be well qualified, and the environment cited as strong … so not total discouragement. And, for those curious about this point in relation to the R21 mechanism, yes, assume preliminary data were presented (& discussed in the first & third critiques).

From an SRO’s explanation of how COI on a study section is appropriately handled:

“It sounds like you are concerned with conflict of interest rules for member conflicts. … At CSR, member conflicts generally occur if someone has a “significant” role on a project. If a member is participating as a consultant, this is considered to be an individual conflict rather than one for the entire study section. So, the member would be in CF and would leave the room during the review, but the study section could then review it.

However, once involvement reaches a certain point, the study section itself is in conflict. If involvement exceeds 5% of effort (more than 0.6 calendar months), or if the member is described in the budget justification as participating in the preparation of manuscripts for publication or receiving other tangible benefit, then a line has been crossed and the study section itself is in conflict. At this point, the “member conflict application” is assigned to a member conflict review group or to another appropriate study section if there is one. These situations are evaluated on an individual basis. It is a question of both (1) tangible benefit, and (2) extent of involvement (e.g., would this be considered “this person’s research”, as our Chief says). There is also the important matter of “perception” of conflict; this is always a matter of judgment by the SRO for the group doing the review. We are to avoid actual conflict of interest, as well as the perception of conflict.”

So you have that priority score that is frustratingly close but no cigar. In fact, your program officer may have held out hope for last minute Council support … but no. What to do?

Case Study:

R21 receives a priority score of 138, 11th percentile (think about this a moment folks - 138 … 11th percentile?) … concerns are incredibly minor, including a quibble over word choice (discussed!). Think, use of sex vs gender appropriately. Not relevant to the science at all. However, this A0 is not funded.

The PI takes the Neuro-conservative zen view on resubmission (First, Do No Harm), including a clean, simple apology for the errant word usage. Remember, this is an R21 with only a 1-page introduction.

The A1 receives a priority score of 107, 0.1st percentile, with the following comments in the summary of discussion: “The proposal is significantly bolstered by the thorough responsiveness to minor concerns from the previous review and with the addition of new preliminary data. … the committee was unanimous in their high enthusiasm for this outstanding application.” Obviously.

So, the PI kept collecting data, did not change anything in the Specific Aims or, that I can recall, in the Background & Significance … did streamline one set of experiments thanks to a recent innovation in the lab.

In this case, I don’t think the program officer was a fount of insight, but I would recommend starting there if not enough guidance for the resubmission is communicated in the resume & summary of discussion. Hopefully your program officer heard or heard of the discussion and can offer pointers for what to address. I’ve seen scores of summary statements for just about every grant mechanism and IC, though, and usually the opening discussion paragraph (for scored applications) throws the PI a bone for preparing an amended application.

Certainly a pristine, laser-focused introduction sans color commentary is critical. Citing new reports in the literature as appropriate (especially by study section members) and of course tying in new data (even better, a new manuscript accepted/published) you have will show reviewers you are committed to seeing this work through in a sound, scientific manner. Appropriately integrating new data from the literature and/or your lab in the research design and methods (additional rationale for your approach, consideration as an alternative approach, insight into potential pitfall or divergence in outcome, etc.) will further increase reviewer enthusiasm. If your IC (institute/center) has a new strategic plan or programmatic priority in line with your proposal, point this out as well. The key is not to open yourself up for potential target practice with the insertion of a new theory or premise or methodology. And, of course, not to get in a pissing match with the study section.

I’m sure comments from yuns will add a battery of great suggestions. I find it difficult to offer blanket advice on this sort of situation without knowing the application and its history. Grantsmanship instruction is available freely everywhere … the key is to get good input on your application specifically rather than try to fix it in a vacuum, no matter how many grant-writing workshops you’ve attended or Websites you’ve clicked through. An army of one was never less likely to succeed as in the battle for grant funding.

Okay … a little help here, and this is no April Fool’s. I need to come up with 45 min worth of what not to do if you want to get funded. I could come up with dozens and dozens of bullets like this, and I have priceless gems from actual narratives that I couldn’t possibly make up if I tried. But I really am interested in what fatal flaws crop up in the routine grant-writing life of a reagent-quality scientist. When I’m all done in a few weeks, I promise to slap it up as a resource page (like NIH Paylines & Resources).

- Let the NIH assign your grant
- Mislabel tables/figures (better yet, don’t include any)
- Make legends small/cryptic (taking up valuable space)
- Don’t bother with letters of support/collaboration
- Only address the comments you agree with in your revision introduction
- Propose experiments using reagents/cell lines/transgenic models you don’t have & are challenging to generate
- Propose complex aims each of which is contingent on achievement of the one prior
- End vague sentences describing statistical analyses with “etc.”
- Mention by name as many well-known investigators as possible without explaining/justifying their role, which has no budget allocation either (no need for biosketch, everyone knows who they are - or a letter of support, hate to bother them)
- Use as many uncommon abbreviations as possible (defining each is optional - saves more space)
- Don’t waste space between paragraphs
- Remember, instructions are for sissies
- Never, ever show your grant application to anyone before submission

No, I’m not providing such … this was the conference I attended today. Needless to say, one full day wasn’t near enough, but when you don’t have industry sponsorship … The endowed lecture, this year entitled “How Financial Conflicts of Interest Endanger Our Profession”, was given by Jerome Kassirer, MD (Prof of Med, Tufts SOM; Visiting Prof, Stanford SOM; Editor-in-Chief Emeritus, NEJM). An equally fascinating keynote address was delivered by George Lowenstein, PhD (Herbert A. Simon Prof of Economics and Psychology, Dept of Social & Decision Sciences, Carnegie Mellon) on “A Psychological Perspective on Conflicts of Interest in Medicine.”

There was a consensus on the failure of disclosure as a means to remedy COI, despite its perception as the “great disinfectant.” Dr. Lowenstein presented a series of, yes, elegant studies demonstrating motivational information processing (brain processes information differently depending on whether processor wants to embrace or discard the information presented), that physicians are willingly misled even when they know the information being presented is biased, and that disclosure of a COI perversely corrupts the behavior, decisions, & actions of both the discloser and the disclosee. Dr. Kassirer added that disclosure alone requires an assessment of motives (difficult if not impossible) and gives the discloser license to proceed “unconflicted” in an “anything goes” mode.
Read the rest of this entry »

Two items in Nature journals consider the benefits of a consortium approach to peer review of journal manuscripts (in existence) and grant applications (modest proposal). Nature Neuroscience announced today it is joining the the Neuroscience Peer Review Consortium, which “reduces the overall reviewing workload of the community by allowing authors to continue the initial review process when their paper moves from one consortium journal to another, once the paper has been rejected or withdrawn from the first journal.” The Nature Neuroscience editorial describes the process, voluntariness, and flexibility and notes that the NPRC system will be evaluated at the end of the year … and on an ongoing basis at the journal’s blog, Action Potential.

Separately and quite distinctly, in a letter to Nature, Dr. Noam Harel of Yale makes a modest proposal: a centralized grant proposal repository into which applications could be deposited at the PI’s leisure and that sponsors could search for interesting science to review and possibly fund (no doubt with some encouragement by depositing PIs). The research proposals would only be made available to sponsor agencies, and multiple sponsors interested in the same work could collaborate on a shared funding agreement. As a thought exercise, interesting. As something to actually implement …

And finally, while we’re pondering peer review, Gregory Cuppan, a managing principal at McCulley/Cuppan (which specializes in document development), contributes to the commentary on a prior thread discussing the Publishing Research Consortium survey data. Specifically, he notes that “most people have little or no formal training in the task of review” and would “be interested to know how many readers of this blog have actual formal training in the task of review (here I make a strong distinction from training for the task of editing).” He refers to a 1961 study by the Educational Testing Service in which 53 distinguished reviewers read 300 college student papers but only had a median correlation among reviewer scores of 0.31.

In a separate note to me, he also suggested we look at an article by Mayo et al. in the Journal of Clinical Epidemiology suggesting traditional grant review processes and funding decisions suffer from a high degree of variability due to too few reviewers being involved. The report presents empirical data from intramural review of pilot project applications at McGill University Health Center Research Institute; applications were both ranked and scored (1-5 scale), with poor agreement between the two (kappa value of 0.36, with 95% CI 0.02-0.70). The top-ranked proposals would have failed to meet the “payline” with varying probability depending on who was assigned to provide a scored review. The examined process does not translate to current NIH study section practice, but it lends credence to the recommendation (see pp 4-5 & 38-41) that chartered study section members (not ad hoc reviewers) rank scored proposals at the end of each meeting. Per Mr. Cuppan’s suggestion that manuscript reviewers lack training, see pp 45-46 of the Enhancing Peer Review report for standardizing reviewer, chair, and administrator (officer) training.