Journal Peer Reviewers: Incompetent & Biased

So reports an article by intramural researchers at NIEHS in this month’s issue of Science & Engineering Ethics entitled, “Perceptions of Ethical Problems with Scientific Journal Peer Review: An Exploratory Study.” The authors – David Resnik, Christina Gutierrez-Ford, and Shyamal Peddada – conducted an anonymous survey of participants at mandatory RCR training sessions at NIEHS (51% response rate). The average age of respondents was 42, with an average of 35 papers published.

Here were the questions and percentage of “yes” responses:

Have any of the following ever happened to you during the peer review process?

    A reviewer was incompetent – 61.8%
    A reviewer was biased – 50.5%
    A reviewer required you to include unnecessary references to his/her publication(s) – 22.7%
    Comments from reviewers included personal attacks – 17.7%
    A reviewer delayed the review so that he/she could publish an article on the same topic – 9.6%
    A reviewer breeched confidentiality – 6.8%
    A reviewer used your ideas, data, or methods without your permission – 4.5%


Not surprisingly, there was a significant age effect with regard to perceptions about personal attacks: “the older the person, the more likely he/she would claim to have experienced at least one personal attack in his/her research career.” Similarly, postdocs (n=94) and principal investigators (n=38) were more likely than staff scientists (n=55) and technicians (n=33) to report negative perceptions of peer review. No differences were noted in the perceptions of men (n=119) versus women (n=97).

These are, of course, perceptions that cannot be verified to occur in the peer review process. As the authors note, how would a researcher know whether a reviewer breached confidentiality? Still,

documenting that scientists perceive that there are ethical problems with journal peer review can be an important finding in its own right, because a scientist may change his/her behavior in response to what he/she perceives to be a problem.

The authors urge others to study perceptions of peer review both to confirm the need for reform and to guide these reform efforts. For example, since these survey respondents reported incompetence and bias to the biggest problems, the authors suggest that

journals, research institutions, and scientific societies should consider ways of dealing with these problems, such as providing additional education and training for reviewers on the scientific and ethical standards for peer review, requiring reviewers to disclose conflicts of interest, and paying more careful attention to the selection of reviewers.

Nothing new or radical here, though they do then note that open review might need to be considered to address the most serious reported violations of peer review ethics, such as breach of confidentiality or stealing intellectual property.

12 Comments »

  1. BB said

    So sad. Maybe the open-access non-anonymous approach by Biomed Central ought to become the norm and not the exception.
    But what to do about the grant review process?

  2. George Smiley said

    A reviewer was incompetent – Oh, yeah.

    A reviewer was biased – Yup.

    A reviewer required you to include unnecessary references to his/her publication(s) – Tried to, on a paper currently in the cycle. Won’t succeed.

    Comments from reviewers included personal attacks – Yup. Happened last year. Idiot editor refused to deal with it, too. Resubmitted to a better journal and got it published. Bite me, asshats.

    A reviewer delayed the review so that he/she could publish an article on the same topic – Not to my knowledge. I think that people making such a claim are generally inferring more than they can know.

    A reviewer breeched confidentiality – Not to my knowledge.

    A reviewer used your ideas, data, or methods without your permission – Not to my knowledge. I think that people making such a claim are generally inferring more than they can know, and that their thinking is likely less original than they think it is. Mediocre minds think alike.

  3. drugmonkey said

    What GS said about inferring the unknowable. Sitting on reviews while rushing out a manuscript does happen, but less (far, far, far, less) than it is suspected to have occurred.

    And yes, ideas are indeed cheap. It does not have to be mediocre minds that arrive at the same way to address what is, after all, a scientific problem that is open to all in the field for consideration.

  4. PhysioProf said

    What a fucking joke! This is about as useful as asking baseball pitchers how frequently they think home plate umpires unfairly call their strikes as balls.

  5. whimple said

    Mandated annual salary increases combined with frozen budgets at NIEHS have resulted in zeroed-out supply budgets for the intramural labs, so I guess this is better than sitting around reading blogs all day to keep NIEHS intramural researchers busy.🙂

  6. George Smiley said

    Drugmonkey — my comment about mediocer minds thinking alike is simply to point out that the brighter you are, the more likely you are to realize that there are other smart people out there, and that some of them are your competitors. It’s the less bright folks — the ones who, perhaps, struggle to come up with ideas — who think their ideas are being stolen.

  7. drugmonkey said

    Now that is a very interesting take, GS.

  8. George Smiley said

    “Very interesting?” I hope you’re not trying to steal my idea, dm. I’d hate to think that. But it sure looks that way. After all, who else would have thought such a thing besides moi?

  9. In defense of reviewers said

    PhysioProf hit the nail on the head

  10. […] they decide within 1-2 days whether to accept the paper for review. If reviewers are flagged as incompetent, I’m sure these authors rejected out of hand have some choice descriptors for the editors […]

  11. […] by ErroR0390 on Tue 30-12-2008 The failure of open science Saved by fungogh on Sun 28-12-2008 Journal Peer Reviewers: Incompetent & Biased Saved by funkymcfunk on Fri 26-12-2008 Media News Catch-Up: 8/28/08 Saved by HouseholdOpera on […]

  12. writedit said

    A peek into how authors respond to these incompetent & biased reviewer suggestions:

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: