Opinions Sought on Review Process – RC1 et al.

Last week, a commenter on the RC1 thread asked for reflections on the two-stage peer review process, particularly the Editorial Boards:

Were you on an Editorial Board? I’d still love to hear more feedback on what the reviewers thought of those.

In fact, I’d love to hear more feedback on what applicants and reviewers felt about the Editorial Board review process in general. Independent of the huge numbers, short time frames and 1-2% success rate. Would you want regular study sections to use this process?

Other questions of interest to this individual (and others who have contacted me directly) include:

Was it rigorous? Did it seem like a waste of time? Were the scored apps of high quality? Did you feel rushed? Do you feel that better science got funded by the Editorial Board review process than if grants were picked by random lottery? Did the second level (after mail review) add anything?

Were new/unknown investigators at a disadvantage? Was science outside the interest/expertise of the Editorial Board members at a disadvantage?

I know some questions have come up about conflicts of interest among reviewers, which the NIH recently addressed in its Challenge Grant FAQ:

How were conflicts of interest managed for the Challenge reviews?
Given the volume of applications received and the compressed timeline for finishing the reviews, the NIH determined that it was necessary to recruit over 15,000 outstanding scientists to serve as mail reviewers (including some who would also be applicants). However, a Challenge applicant could only serve in the Challenge reviews as a mail reviewer and not as a study section member, and only for a study section(s) other than the one reviewing his/her application. Mail reviewers do not participate in the discussion or final scoring of the applications, and do not interact with other study section members.

Hmm. Except Editorial Board reviewers were asked to score applications based on the mail reviewer scores and critiques … though apparently most Editorial Board members felt they could not do so without looking at the original application … often leading to critiques of the mail reviewer critiques … and so on.

And heck, why stop at the special process used to review RC1s …. How do reviewers (and, I suppose, applicants) feel about the new review, scoring, and critique procedures?

One Editorial Board member told me that on more than one application, the mail reviewers had very divergent scores but were in agreement with their critiques/opinions, suggesting the learning curve will be steep on the uniform assignment of scores. Perhaps the NIH could use these thousands of clusters of three naive (in terms of the scoring procedure) reviewers looking at the same application to analyze patterns of score assignment against the written comments. I know just the person to write the grant application to fund this …

And what about the plan for increased use of alternatives to in-person study section meetings, which is when many of these finer points would be addressed and, of course, advocates speak out on behalf of specific applications.

Fire away, folks. The NIH needs all the feedback it can get.

9 Comments »

  1. BP said

    “However, a Challenge applicant could only serve in the Challenge reviews as a mail reviewer and not as a study section member, and only for a study section(s) other than the one reviewing his/her application.”

    That’s not right. I was a mail reviewer for a study section that reviewed one of my proposals, and I know someone else who was a mail reviewer for her study section too.

    • hohum said

      Same here. I was a mail reviewer for the study section that my proposal was being reviewed by. I specifically asked whether this was OK prior to accepting the assignment and I was assured that it was.

  2. Challenge grant applications are for suckers who’d rather not spend their time on science or on real grant applications. That was pretty damned obvious from day one. The program was FUBAR from its inception; I just feel sorry for the program officers who are trying to implement this damned thing that most of them did not want and certainly did not ask for.

    • D said

      The only stupid part was describing it as a source of “easy money” and encouraging everybody to apply. I am not sure if Deans and Chairs or NIH Admin is more to blame for this.

      Your pity is misplaced. It is the Review Officers who are suffering for this.

      As for the program officers, challenge grants will add very little to their work load. I mean 200 divided by 20+ ICs is not very many. (Unless NHLBI does pick up 200 extra as has been hinted here). Even adding in the RC2s isn’t going to add much. It is the ARRA supplements that are probably driving them crazy. Most of those have to be reviewed internally. And of course answering questions on the phone when they have no answers.

      The bottom line is that it is not easy to spend 10 giga dollars in a few months.

  3. qaz said

    The information in the Challenge Grant FAQ is wrong.

    Like BP, I was a mail-reviewer on a study section that I had an RC1 in. I specifically asked my SRA about this and was informed that they had run out of reviewers and to do my best to be fair.

  4. Odyssey said

    A colleague here described the whole RC1 business as a denial of service attack on the NIH…

    • D said

      Very funny and so true.

  5. […] The best description of the effect of the ARRA/stimulus on the NIH that I’ve seen so far was provided by our good blog friend Odyssey of Pondering Blather blog. Over at Medical Writing, Editing & Grantsmanship Odyssey observed: […]

  6. sansan said

    As a reviewer and I know many others would agree that the quality of RC1 application was very poor and appears to be a waste of time.
    The ARRA money could be better spend on administrative supplements.

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: