So, the good news is that the scientific community has an excellent representative on the NIH’s program Enhancing Peer Review in Keith Yamamoto, PhD (UCSF). You can see Keith, his intramural counterpart Larry Tabor (NIDCR), and the Great Zerhouni himself in a recent videoconferenced live discussion of the peer review reform initiative. The Powerpoint presentation to the research deans tunining in on this discussion are also available.
Issues raised after the PPT include blinded peer review, external hired reviewers, quality of peer reviewers (longing for the old days with smaller, higher quality panels reviewing fewer applications in more depth – the sort of study sections on which members looked forward to serving), role of publication record of reviewers, striking a balance between experts and generalists on panels, reflecting individual study section member discussion points, the need for special review panels for new investigators, the possibility of capping the number or dollar amount awarded to individual investigators (& possibily institutions), how to avoid bias based on submitting PI and/or institution, and the need to be able to recognize & reward promising new investigators through their track record &, well, promise. Some comments were a bit odd, such as why applications are dinged for environment (um, if they don’t have the resources and don’t collaborate with a group who does have the the appropriate facilities, how do they expect to do the proposed science?).
The Great Zerhouni noted that complaints about reviews being too conservative have persisted throughout the history of peer review and indicated this finding motivated the development of the Pioneer Award. The latter involves shorter applications and a 2-part review process (electronic and then interview) to reward truly transformative ideas, which has proved successful … per Elias. The GZ also confirmed they are considering a shorter pre-proposal submitted for review of innovation-significance of aims followed by invitations for full proposals. This makes excellent sense for the larger mechanisms in particular but could also weed out across the board the not-ready-for-primetime triage candidates with less pain and time investment on both sides.
You will learn that they will be conducting a series of 5 regional town hall meetings through October … that in Feb 2008, NIH “Leadership” will consider input from the RFI and the town hall meetings and the working groups and make suggestions for next steps, including possible pilot projects. Remarkably, the pilot projects will be designed and launched in March 2008 … the cynical among us would conclude that, well, the work on these pilots will be underway before all the data and comments are in and fully ruminated.