CSR Peer Review Notes

The January 2009 edition of Peer Review Notes is hot off the PDF press, though disappointedly lacking in substance considering how many substantive questions we all have about the upcoming spring round of study sections, which will inaugurate, so to speak, the new scoring and review/summary statement procedures. As Jon Stewart might say, “Ewwwwh, the new scoring system is based on a 9-point rating scale using only integers.”

10 Comments »

  1. iGrrrl said

    A key issue, pointed out by NIH, is that the new system is likely to produce more “tie” scores. That means program officers and Council may have more discretion about relevance to “mission relevance and portfolio balance.” NIH also notes that they’re concerned about inter-rater reliability, and the narrower the scale (9 possible ratings vs 41) the better the convergence of scores.

    I also noted that they’re trying the “editorial board” approach with the TR01s. This idea came up several times during the early phases of the Enhancing Peer Review process. Any thoughts on this?

    The editorial board approach has been discussed in recent presentations as well (e.g., Scarpa & WIlder at the Nov 3 PRAC meeting), and the T-R01 Website itself notes the specialized review process. I think it will only be workable for special mechanisms with focused objectives (and in those cases I think it could work well), though I suspect the process will be subject to the same complaints (by unsuccessful applicants) as those directed at elite journals in terms of the editors and editorial boards involved. It seems one person to ask for feedback on how well the editorial board approach worked might be the SRO for the BRP program. -writedit

  2. whimple said

    The system *should* produce more tied scores — that’s a good thing. There’s no point in evaluating applications to three significant digits if the evaluation is incapable of that level of meaningful discrimination; all that does is give the illusion of rationality to an irrational process.

    Excellent synopsis of the problem at hand, whimple. I have yet to find a reviewer who can distinguish between applications scored at 1.49 and 1.52. And of course, program always had plenty of discretion in making awards, and making it clear they are deciding among essentially equal applications should, theoretically, help applicants understand that their score is not the final answer in terms of funding. – writedit

  3. bikemonkey said

    Yup, tied scores are a feature not a bug. Somewhere in the announcement or rationalization on the PRAC website they made it pretty clear that part of the problem was a false confidence that a 148 grant was actually more meritorious than a 152 grant. Actually, wasn’t that in the decade-old review from the last round of low-budget breast beating over supposedly broken review?

  4. D said

    The problem with tied scores is that it is much easier to fund Dr. Famous (who might have a director’s ear) than a early stage investigator (who might not have the gumption to harass a program officer). In other words, at least in some institutes, I bet ties go to the more famous and already well funded.

    It will be interesting to see how it pans out. I can’t wait to see Program’s SOPs on how these decisions will be made. Talk about potential conflicts of interest.

    D

  5. D said

    I think most SROs and POs are also curious about how these new systems will work. Not much (actually no) guidance has been provided to the implementers yet.

    D

  6. BB said

    What about PO’s who refused to be “harrassed?” Ever run into those? I have.

  7. D said

    Yes. But, not many. The larger problem is how you tackle conflicts. Let’s say a PO has 2 grants that get a 1.8. You can only fund 1. One grant is from a PI that has been in your portfolio for 9 years. You see them at meetings, have an occasional beer together, talk on the phone, have wathced them progress from young asst. prof to tenured assoc. prof., etc. Not really friends but you clearly have a history. The other grant is from a new PI from a lab you don’t know very well who is just a “number” to you. How do you decide who to fund? These kind of conflicts exist. It is human nature. What criteria do you use to decide?

    This is where I foresee problems. Be careful what you wish for. You might get it.

  8. whimple said

    Let’s say a PO has 2 grants that get a 1.8. You can only fund 1.

    Give them both 2 year R56 awards, and keep the extra year of funding for someone else.

  9. D said

    A very Solomon-like proposal. Unfortunately, he doesn’t work at NIH.

    D

  10. drugmonkey said

    The problem with tied scores is that it is much easier to fund Dr. Famous (who might have a director’s ear) than a early stage investigator

    I think more generally this is to free up Program from the tyranny of the percentile rank. They already have a great deal of latitude but this will encourage more of it.

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: