So asks the cover story by Meredith Salisbury for Genome Technology.
She starts the ball rolling in her accompanying editorial:
Get three scientists together, and it’s almost a guarantee that the conversation will eventually turn toward the vagaries of the peer review process. Be it for winning grant funding or getting a paper published, this system of relying on a handful of fellow scientists to select the most promising and influential research shapes — at least to some degree — every single researcher’s career path.
And then she gets 3+ scientists together, with the tone set right off the bat by Ferric Fang:
“For something that is of and for scientists, the peer review process is very unscientific,” says Ferric Fang, a professor of laboratory medicine and microbiology at the University of Washington. Whether it’s for papers or grants, having just a handful of people review someone’s work is statistically unsound, he adds. “If these [reviews] were data that you generated in your lab, you would say, ‘I don’t know what the conclusion of this is.'”
And per the suggestions made on grant review processes, apparently efforts to enhance peer review at the NIH haven’t gone far enough. A sampling to get you over to Genome Technology for the full report:
One hope is that having a larger pool of reviewers could help reduce the impact of any individual review, says Fang. Under the current system, “one bad review can sink an application.”
Another take on the grant review system in general is that focus needs to shift away from today’s model of specific proposals for short-term periods. …
Lawrence [Peter Lawrence at the zoology department of the University of Cambridge] would prefer a system where reviewers considered the track record of the investigator more than the details of the new research proposal (with special dispensation for new investigators). …
According to Fang, this concept of awarding funds on a track record basis would also serve the purpose of weeding out people who are very skilled at writing proposals but are less competent at actually performing the science.
Oh no! What’s a writedit to do?! Well, I am the first to acknowledge that no amount of skilled grantsmanship can make up for poor science, so I think, to a certain extent, this last concern can be dispensed with.