Journal Retraction & Double-Blind Review

First, an early casualty of Deja Vu as reported in Nature: “A review article written by a rheumatologist at Harvard Medical School in Boston, Massachusetts, has been retracted after the journal, Best Practice & Research Clinical Rheumatology, learned that more than half of the paper may have been plagiarized. The 2004 article, by Lee Simon (Best Pract. Res. Clin. Rheumatol. 18, 507–538; 2004), was manually checked after surfacing in an automated trawl through 7 million biomedical abstracts for possible plagiarism (see Nature 451, 397–399 ; 2008). The retraction was announced on 29 January. Harvard Medical School has formed a committee to review the matter but has not launched an official investigation, says spokesman David Cameron. Simon declined to comment, saying only: “I’m very sorry that I’ve been so targeted for something like a review article.””

This week’s Nature also includes a commentary on double-blind peer review of journal articles drawn from the Publishing Research Consortium survey results that “also highlight that 71% have confidence in double-blind peer review and that 56% prefer it to other forms of review. Support is highest with those who have experienced it (the humanities and social sciences) or where it is perceived to do the most good (among female authors). The least enthusiastic group is editors.”

Considering that so many manuscripts include referenced comments such as, “We previously showed …”, I have to wonder how blindable journal articles can be. Indeed, as Nature notes, “The editors at the Public Library of Science abandoned double-blind peer review because too few requested it and authors were too readily identified.”

Nature invites your comments on this commentary.

12 Comments »

  1. Alexander Shearer said

    We were discussing blind reviews just this week with members of a number of other research groups in our field. Apparently one method used to avoid the most obvious self-identification involves “anonymizing” those types of citations, replacing “We previously showed…” with “Blah et al previously showed…” Assuming acceptance, these citations are then reworked into self-referential syntax in the final resubmission of the paper.

    Even so, it’s usually apparent when there’s strict continuity between a paper and the work it’s citing, such that it must necessarily be coming from the same people.

    Interesting to hear this is a topic of conversation out there. I thought about the anonymizing bit, but wondered whether the burden would be placed on the authors (who would not be thrilled with this task and might miss some referents) or on the journals (who likely would not be enthusiastic about hiring additional copy editors to manage the anonymizing/de-anonymizing process). Thanks for chiming in to share your discussion with us. – writedit

  2. CC said

    Considering that so many manuscripts include referenced comments such as, “We previously showed …”, I have to wonder how blindable journal articles can be.

    To me, the most troubling concern about unblinded review is unconscious or semi-conscious sex discrimination. Even trivial blinding should protect the first author, and in most cases everyone, on that front. If a simple measure can address that problem, I’d prefer that to an elaborate solution to stop the much rarer issue of active malice.

    Indeed, as Nature notes, “The editors at the Public Library of Science abandoned double-blind peer review because too few requested it and authors were too readily identified.”

    It seems plausible that people are reluctant to *request* it, for fear of seeming paranoid and defensive, but wouldn’t object if it were mandatory. Or maybe the vast majority of scientists consider themselves in the category who benefit from the reviewers know who they are?

    I absolutely hear and concur with your concern. So, eliminate everything on the manuscript cover page supplied to reviewers except the article title (to prevent potential bias based on institution) and assume the reviewer won’t be so motivated as to look up references for full names and affiliations? Though it wouldn’t be true blinding, I’m not sure why editors would object to making this simple change a universal standard. With grant applications, reviewers need to be able to confirm the available environment/facilities will support completion of the proposed work and that the PI has the right skill set. With journal manuscripts, however, the work is done, and reviewers should focus entirely on the science … not the whodunnit. – writedit

    P.S. Be sure to check – and contribute to – the excellent discussion on this topic at Nature, including special consideration of the effect of double-blind peer review on gender bias.

  3. whimple said

    With grant applications, reviewers need to be able to confirm the available environment/facilities will support completion of the proposed work and that the PI has the right skill set.

    Not necessarily. The reviewers could just come up with an assessment of scientific merit to a project, and then let program worry about whether the PI and / or environment are up to the task. The Army reviews some of their Congressionally Directed Medical Research Program grants fully blinded this way.

    The NIH is considering dropping the “environment/facilities” criterion from peer review in large part because it generally don’t have much relevance in the review of proposals.

  4. bikemonkey said

    Careful there Whimple! You are going to get us talking about the fate of our overhead $$$. Can you imagine if the Program analysis of the environment/PI included manipulating the IDC? “Ah, see here BigU, you don’t appear to actually be supporting the junior PI we’d like to fund so we’re going to dock your overhead and put it into the direct budget. Oh, and don’t bother with your usual shenanigans of taking it out of the jr PIs hide anyway because we’re all over that action like flies on…”

    BM, I’m surprised you didn’t jump in with a tirade against giving Program more leeway (unrestricted by priority scores) in funding decisions, as could occur if they could point to real or imagined strengths (if it’s someone they want to fund no matter what)/weaknesses (if they need the $ for a PI in the prior category) in an applicant and/or his/her research environment when making those vicarious funding decisions. We can watch for this thread over at DrugMonkey.

  5. Maxine said

    Thank you for the very nice write-up, much appreciated. I personally think double-blinding peer-review is hard to achieve in practice: as you point out, the identity of the authors is pretty obvious without names being listed (the reference list is usually a giveaway!). When we interview candidates for new editor positions, usually people doing postdocs themselves, we delete the author names from a typical submittted paper (or two) and ask the candidate questions about it or them. Almost invariably, the candidate spontaneously identifies the authors.
    We go to much effort to ensure that the peer-review process is fair, as discussed on our “ask the editor” forum on Nature Network (http://network.nature.com if you have time to take a look). The system probably has its flaws, as does any system, but I don’t think that blinding reviewers (or trying to) would be a panacea.

    Thank you so much for taking the time to comment here, Maxine … and thanks so much more to all of you at Nature publishing for raising such thoughtful questions and launching spirited public debate throughout the community of science. I’m chewing over a couple of items in Nature Nanotechnology right now, actually. – writedit

  6. whimple said

    Question for Maxine:

    The biggest hurdle to overcome in publishing in Nature is getting the editors to agree to send the manuscript out for review. Has Nature ever internally done the experiment of seeing to what degree who is on the author lines affects which manuscripts are sent for review?

    You might want to try this at the aforementioned Ask the Editor forum (full URL here). -writedit

  7. bikemonkey said

    not sure how this would help whimple. so the editors send out more stuff and still make the decision to accept the big name paper and reject the unknown-authors paper given approximately equal reviews. what’s been changed?

    it would be more productive to address the circularity of what constitutes “a C/N/S paper”, would it not?

  8. whimple said

    Having the material sent out for review at least gives it one more chance to be evaluated on the merits rather than the personalities. I wonder how often the editors look at a paper and think, “seems good, but we’re really not sure what to make of this” in the initial evaluation and whether or not the names on the author line helps push this first impression one way or the other.

    The other plus for the journals is that if anonymous editorial pre-review is mandatory that gives the editors some good butt-cover for rejecting the odd piece of work from big-name PIs that just isn’t quite up to the big-name PI’s usual standards. The editors can use, “oops, sorry, we didn’t know it was you!” as their excuse so as to mitigate the piss-offedness of big-name PI so that future (up to snuff manuscripts) will still be sent the journal’s way.

  9. maxine said

    One issue with the “total” blinding suggested here is that the peer-reviewers have to be able to see how the work has built on previous studies — science is not done in a vacuum. The peer-review process cannot identify fraud, it was not designed to do that. If a peer-reviewer is sent a paper with not a single author clue, then how can that reviewer judge the advance against a body of work, compared with someone (and I can tell you, there are plenty of people like this) who would plausibly make it all up? This is far from saying that a peer-reviewer would only take a ms seriously if it is by a senior or famous author. Authors have almost invariably presented their work at meetings, for example, and community discussions have taken place about it, so the referee has some context for reading it. A reviewer cannot assess a manuscript in isolation, and it is important for reviewers (and there are two or three independent ones for each ms) to be able to judge the work in context. And the editor is also assessing the reviewers’ comments, providing a neutral approach — why should an editor “want” to publish a ms by a “famous” author if the ms is boring? They just don’t! They want to publish the exciting work.

    Nature has no problem with rejecting manuscirpts by Nobel laureates and all points from that. We have done so, on many many occasions. We are just not interested in only publishing big names, we are interested in publishing exciting science, whoever it is by.

    As an editor for many years, I am aware that the publication process that is seen by the world is the tip of an iceberg. Many authors behave in any way they can to get their manuscripts published, including “charm offensive” and downright nasty bullying, as well as appealing against every possible judgement made by editors and peer reviewers at every step of the process. In more than 100 years of publishing, have you ever seen or heard a Nature editor making public a case of a paper being considered by the journal, and how the named authors behaved? No. So why an author would worry about asking for their manuscript to be blinded is just not plausible, to me, as the facts demonstrate the journal’s commitment to confidentiality. I have seen authors behave in a way that toddlers (two year olds) would be ashamed of, many many times. I have read words written by authors about their experiences which are the opposite of what actually happened — yet journal editors do not publicly contradict, as these matters are confidential, and we keep that trust which we state as part of our policy.

    At the end of the day, any system of peer review is going to have its critics, particularly for a journal like Nature that rejects so many manuscripts. We think our system is fair and that it works, for all the reasons explained in our peer-review policy pages, the Nature Network forum and in various editorials and so on. We certainly interact with the community and want to hear people’s views, but many of them seem to boil down to unsupported assertions or expressions of personal disappointment or personal opinion — even, on occasion, paranoia. One point that I’ve also raised elsewhere is that “junior” reviewers tend to be far harsher and provide screeds of detailed, often extreme, criticism, compared with “senior” reviewers, who often have more of a sense of perspective. As do editors – at Nature journals, certainly.

  10. drugmonkey said

    One point that I’ve also raised elsewhere is that “junior” reviewers tend to be far harsher and provide screeds of detailed, often extreme, criticism, compared with “senior” reviewers, who often have more of a sense of perspective. As do editors – at Nature journals, certainly.

    I’m going to try to be polite here. forgive me if I fail because it is not exactly personal.

    your sense of “perspective” is in some cases valid. true. but in some cases it is entirely circular. you disproportionally respect the more senior people’s opinions so therefore their opinion becomes your objective standard. topic X is the “best” or “hottest” science because the people doing it, and btw, publishing in your journal and reviewing for you, say so. can you not see this inherent circular approach?

    the fact of the matter is that I can page through issues of Nature (and C and S, this is not specific to y’all) and run across figures in which I cannot tell what the damn error bar is, what the stats are, what the analysis really is. I was just helping someone the other day in interpreting some data where I went hammer and tongs at the analysis, the supplements, the references and it was still not possible to know what in the hell the data really showed. A classic case where a “junior reviewer” may very well have levied “detailed screeds” as to why the data were sheist without this info. A classic case where “senior perspective” would’ve ‘trusted’ their peers. I don’t get it because as far as I was concerned the whole point depended on an effect which had. not. been. demonstrated. by. the. data!

    re: authors’ “charm offensive”. I appreciate your restraint and it is appropriate. I have very little doubt (and indeed some direct knowledge) of authors behaving badly. I also have some reasonably direct knowledge of where editor manipulation works. repeatedly. and this is why I ask editors such as yourself to come out and declare unequivocally that tantrums and stroking does not work. That it has never worked at your journal and that you have specific procedures in place to make sure it canot work.

    And then we can start to ask if this is so, why do so many authors seem to think that it does work.

    After all, if you really dislike author tantrums, wouldn’t making it emphatically clear that as soon as they start that crap it is bye-bye acceptance fix the problem pronto?

  11. […] Filed under Biomedical Research Ethics, Biomedical Writing/Editing Update: We already have an article retraction resulting from use of this database, and both Deja Vu and the underlying tool, eBlast, were […]

  12. […] on the recent Publishing Research Consortium report. Letters to Nature continue the discussion of double-blind peer review (pro, compromise, unworkable, need better data), as does the accompanying online discussion forum. […]

RSS feed for comments on this post · TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: