Peering into peer-review

Vaccines do not cause autism. Yet when one peer-reviewed paper suggested otherwise, on the basis of results that were both wrong and fraudulent, there were terrible consequences. It launched 15 years of unnecessary studies to confirm what we already knew – and sowed confusion, eliminated rational debate about the merits and risks of vaccination, and contributed to a public health crisis.

MS is not caused by constrictions in veins in the neck or back.  Yet one study, on the basis of a pitiful hypothesis and “too good to be true” results, made such a conclusion in a peer-reviewed paper. Not surprisingly, the results could never be repeated,. Yet the paper raised false hopes for MS patients, led to hundreds of unnecessary interventions, and to dozens of follow-up studies that wasted millions of dollars.

Both of these situations were catalyzed not by the scientific community, but by the media’s over-reaction to papers that should never have been published in the first place, but that were somehow accepted in a peer-reviewed scientific journal. Peer-review is one of the mechanisms that the scientific and medical communities use to self-police – hopefully ensuring that audaciously wrong science does not get cemented in the global literature. However, as the above examples exemplify, faith in the peer-review system is greatly exaggerated. This has significant implications for public policy and for social media-mediated medical or scientific discussions.

The engagement of Canadians in social-media discussions about medicine and science is a fantastic development. Unfortunately these discussions are rarely rooted in the best science.  Too often, they are dominated by non-experts who use the forums to advertise biased scientific positions that are not supported by the evidence.

How do these non-experts gain credibility? By cherry-picking supportive papers or sentences from the peer-review literature, and creating a false impression of scientific “debate”.

If their views are wrong, how can they find scientific papers to support them?  Because the peer-reviewed scientific literature provides a veritable buffet of bad science.

More than 900,000 of the 1,000,000 peer-reviewed research paper published each year are irrelevant, poorly designed, or redundant, and are never cited. This means that not a single scientist in the world judges those papers worthy to read and reference.

Many papers in the “highest quality” journals also report studies that cannot be reproduced.  Generally, these papers are not “wrong” per se, but the experiments are “true” only in the narrow conditions under which they were performed.

And now we have fly-by-night, predatory “peer-review” on-line scientific journals that offer to publish complete junk – for a fee.  These journals have no credibility in the scientific community, yet their papers are still anointed with the peer-review moniker, and can be used to fuel misinformed “debate” in social media.

The ease with which complete rubbish can find its way into the “peer-reviewed” literature was highlighted recently by Tom Spears, a science-savvy journalist in Ottawa.  He construed a scientific “paper” from random cutting and pasting of sentences from papers in geology and medicine, merged them into a gobblygook jumble of ideas, and submitted the garble to 18 on-line “peer-reviewed” journals. Half were accepted immediately (for a fee, of course).  A similar “sting” of hundreds of journals was reported last year in the journal Science.

And what is the impact on scientific discussions in the social media? Unfortunately, with over 900,000 papers published each year of questionable quality, it is almost always possible to cherry-pick studies to create a “peer-reviewed, evidence-based” narrative that supports any scientific view one wishes to advance, however erroneous, and be it on climate change, alternative medicine, nutrition, vaccination, evolution etc.

While it will never be possible to stop people from creating narratives to support their biases, it is essential to inform the public that peer-review is but a single criterion with which to assess the value of a publication.  We also need to remember that conclusions from even the best peer-reviewed papers should be treated with extreme caution until they are supported in high-quality papers by multiple independent lines of evidence.

So what to believe?  This is the role of web sites such as this one, and efforts like the Cochrane Collaboration, in which experts work to sort the wheat from the chafe in the scientific literature, examine the body of evidence and present their conclusions to the public in an unbiased way.

Unfortunately, this process does introduce “expert opinion” between the public and the data, but with more than 9/10 peer-reviewed publications either meaningless or full of bunk, and charlatans willing to exploit this fact to their own ends, I can’t think of an alternative.

Aled Edwards is the CEO of the Structural Genomics Consortium, Professor in the Banting and Best Department of Medical Research at the University of Toronto and a Visiting Professor of Chemical Biology in the Nuffield Department of Medicine at the University of Oxford .

Leave a Comment

Enter the debate: reply to an existing comment

  1. InfoWadingRoom

    I 100% love this piece, it speaks to me as an informaiton professional. We Medical/Health Librarians deal with this everyday! They are your BEST partner when it comes to quality information and curating to good from the bad – I wish I could shout from the rooftops “this is what we do!!!” 🙂 Thanks for putting the spotlight on this

  2. Jim Woodgett

    Nice piece Aled but also contains some contentious points. You are absolutely right about the predatory journals but most reputable scientists can sniff out these fake journals within seconds. We should not tolerate them. Tom Speers piece didn’t reference John Bohannon’s sting in Science when it was first published in the Ottawa Citizen (he corrected the omission when it was pointed out). Pot calling out the kettle?

    Who is publishing in predatory journals and why? In other words, these “journals” rely on unscrupulous scientists who can’t publish in reputable journals. The study on the low number of cited papers has been shown to be quite flawed but the message is still true – that we publish too much material and a lot of it simply doesn’t warrant publishing. However, it’s a free world and we can’t tell people not to publish. Instead, we have to have means to highlight what is worthy from what is utter crap – hey – lets call this quality control system “citations”!

    As to alternatives, we don’t need to throw out the whole system of publication of science which has served society well for hundreds of years, we need to modernize it. It is good that communication has been democratized. But just as press barons usurped the freedom of the press, there are elements that will exploit social media and pervert the communication process and prey on ignorance. Part of the answer, at least, must be through education and getting scientists to call out garbage instead of just ignoring it, as has been the tradition. There are also new initiatives in post publication review (PeerJ, PubMed Commons, F1000, etc). Fight rubbish with data!

    What the scientific community need to be care about, is to be seen to be self-appointed experts, an elite that is solely entitled to certify what people should read. Instead, the onus is on this community to make its case each time and allow people to make up their minds. This is a problem when there is overwhelming evidence for something and a pitiful amount of evidence against (especially when equal airtime is often given to both sides out of “fairness”). But truth wins in the end and there are many times in history when the majority of the scientific establishment was on the wrong side of the argument (usually fighting against one of their own with a radical idea).

    But you’ve done a very good service in shining light onto this problem. That is a start.

  3. Karen Born

    I wanted to pick up on your point about a media over-reaction to published papers. Science and research literacy among the media is a hugely important piece of this complex puzzle that you describe above.

    I completely agree with you that the media over-reacts to papers – but we need to put this in context. Today’s media is pressed to report on the news with less time to research, less expertise (eg. very few reporters these days have a health or science beat) and in shorter and shorter sound bites. The constant pressure on media to produce new content 24/7 means that the results of studies which may not be high quality are being overhyped by media and issues are treated in a sensationalist rather than cautious way.

    We have a number of terrific health and science journalists who try to fight the pressure to write sensational headlines or exaggerate the results of studies and take their responsibility to frame and translate science and medicine in a responsible way very seriously. However, in the current media environment its getting tougher to do that.

    In 2005 PLoS Medicine did a terrific ‘debate’ series that delved into the roles and responsibilities of media in disseminating health care information, which is worth a read for any journalist who will be covering health or science news in their career, and for those of us who care about how the public gets science and health information.

  4. Gerry Goldlist

    Excellent article. I like to put studies in the context of the basic science I already know. If it sounds too good or bad to be true then it probably isn’t true. That having been said, I still try to keep my mind open when studies are repeatable.

  5. Joanne Gaudet

    Thank you for this post. My research has led me to question, however, would fly by night ‘predatory’ peer review exist without (1) pre-publication with its structurally secretive editorial judgements and decisions, and (2) the framing of pre-publication journal peer review with reader-pay business models as dominant? These are important questions and only come about when we start investigating journal peer review as a scientific object of study… I invite you to peruse a few preprints on my research: – Gaudet, J. 2014. All that glitters is not gold: The shaping of contemporary journal peer review at scientific and medical journals. uO Research. Pp. 1-23. – : Gaudet, J. 2014. Investigating journal peer review as scientific object of study. uO Research. Pp. 1-11. – Gaudet, J. 2014. How pre-publication journal peer review (re)produces ignorance at scientific and medical journals: a case study. uO Research.

  6. Joanne Gaudet

    Thought you might be interested in historical pieces that go into more detail on proposed ‘beginnings’ of journal peer review:… – Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part I. uO Research. Pp. 1-24.… – Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part II. uO Research. Pp. 1-20.

  7. Joanne Gaudet

    Sorry about the broken links… and thanks to Vanessa for pointing this out. Here are the correct URLs for historical pieces that go into more detail on proposed ‘beginnings’ of journal peer review: – Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part I. uO Research. Pp. 1-24. – Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part II. uO Research. Pp. 1-20.

  8. Joanne Gaudet

    For insights on ‘predatory journals’ from a journal editor with a post-publication open review model (profitable to boot), here is a presentation he did at the Association of Learned and Society Publishers in 2013 – He also has several published papers on the model, used at 10 journals in the Copernicus suite of journals.

Submit a comment