No Gender Bias in Peer Review

A piece in this week’s issue of Nature by Herbert Marsh (professor of education at the University of Oxford, UK) and Luz Bornmann (PhD student at the ETH University in Zurich, Switzerland) asks (and answers) the question, Do women have less success in peer review? The Naturejobs item is so brief that I include it here, hopefully without raising the copyright ire of Nature Publishing Group.

Peer review assesses what is of value in science, yet it has been widely criticized for biases. One such perceived bias is gender. But evidence for such a bias has been contradictory. A 2007 meta-analysis (L. Bornmann et al. J. Informet. 1, 226–238; 2007; see also Nature 445, 566; 2007) concluded that women are at a disadvantage in peer review. As this study incorporated all known research on this issue, it seemed a definitive answer.

However, a study published last year (H. W. Marsh et al. Am. Psychol. 63, 160–168; 2008) presented conflicting results. It was the most comprehensive primary-research study, based on data from the Australian Research Council (10,023 reviews by 6,233 external assessors of 2,331 proposals from all disciplines). The study found that the gender of the applicant had no effect on the outcomes of peer review, irrespective of the discipline, the gender and nationality of the reviewers, and whether reviewers were selected by a funding panel or chosen by the applicants.

Why should these two studies have conflicting results? To investigate, both research teams worked together to reanalyse the data and extend the original meta-analysis. We applied new, stronger statistical approaches to 66 sets of results representing 353,725 proposals from 8 countries. In this extended study, which will be published in Review of Educational Research, we found no effect of the applicant’s gender on the peer review of their grant proposals. This lack of effect held across country, year of publication of the studies included in the meta-analysis, and disciplines ranging from physical sciences to the humanities.

The study did, however, reveal very small — but statistically significant — gender differences in favour of men for the 26 sets of results that were for fellowship applications. However, these fellowship results varied greatly between the individual studies within the analysis, indicating that they are not generalizable. We suggest that the differences might have arisen because fellowship applicants tend not to have established a solid track record in their research. In the absence of sound evidence on which to base their judgements, peer reviewers might therefore have been influenced by irrelevant characteristics such as gender.

At least for grant applications, all of the co-authors from each of the research teams agree that the weight of evidence suggests that the applicant’s gender has no effect on the outcome of peer review, and that these findings are robust and broadly generalizable.

Of possible interest is the fact that male authors are the ones examining these issues … I wonder if on this side of the pond, Molly Carnes and her cronies might opt to take a look as well.

3 Comments »

  1. D said

    I thought someone did this informally with NIH study section data. Drugmonkey or Comrade?

  2. sahsaj said

    These conflicting results do call for a qualitative assessment (in depth exploration of a sensitive topic). Not that I am biased one way or the other, but it would be good to know and highlight the concerns.

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: