Peering into peer-review

Vaccines do not cause autism. Yet when one peer-reviewed paper suggested otherwise, on the basis of results that were both wrong and fraudulent, there were terrible consequences. It launched 15 years of unnecessary studies to confirm what we already knew – and sowed confusion, eliminated rational debate about the merits and risks of vaccination, and contributed to a public health crisis.

MS is not caused by constrictions in veins in the neck or back.  Yet one study, on the basis of a pitiful hypothesis and “too good to be true” results, made such a conclusion in a peer-reviewed paper. Not surprisingly, the results could never be repeated,. Yet the paper raised false hopes for MS patients, led to hundreds of unnecessary interventions, and to dozens of follow-up studies that wasted millions of dollars.

Both of these situations were catalyzed not by the scientific community, but by the media’s over-reaction to papers that should never have been published in the first place, but that were somehow accepted in a peer-reviewed scientific journal. Peer-review is one of the mechanisms that the scientific and medical communities use to self-police – hopefully ensuring that audaciously wrong science does not get cemented in the global literature. However, as the above examples exemplify, faith in the peer-review system is greatly exaggerated. This has significant implications for public policy and for social media-mediated medical or scientific discussions.

The engagement of Canadians in social-media discussions about medicine and science is a fantastic development. Unfortunately these discussions are rarely rooted in the best science.  Too often, they are dominated by non-experts who use the forums to advertise biased scientific positions that are not supported by the evidence.

How do these non-experts gain credibility? By cherry-picking supportive papers or sentences from the peer-review literature, and creating a false impression of scientific “debate”.

If their views are wrong, how can they find scientific papers to support them?  Because the peer-reviewed scientific literature provides a veritable buffet of bad science.

More than 900,000 of the 1,000,000 peer-reviewed research paper published each year are irrelevant, poorly designed, or redundant, and are never cited. This means that not a single scientist in the world judges those papers worthy to read and reference.

Many papers in the “highest quality” journals also report studies that cannot be reproduced.  Generally, these papers are not “wrong” per se, but the experiments are “true” only in the narrow conditions under which they were performed.

And now we have fly-by-night, predatory “peer-review” on-line scientific journals that offer to publish complete junk – for a fee.  These journals have no credibility in the scientific community, yet their papers are still anointed with the peer-review moniker, and can be used to fuel misinformed “debate” in social media.

The ease with which complete rubbish can find its way into the “peer-reviewed” literature was highlighted recently by Tom Spears, a science-savvy journalist in Ottawa.  He construed a scientific “paper” from random cutting and pasting of sentences from papers in geology and medicine, merged them into a gobblygook jumble of ideas, and submitted the garble to 18 on-line “peer-reviewed” journals. Half were accepted immediately (for a fee, of course).  A similar “sting” of hundreds of journals was reported last year in the journal Science.

And what is the impact on scientific discussions in the social media? Unfortunately, with over 900,000 papers published each year of questionable quality, it is almost always possible to cherry-pick studies to create a “peer-reviewed, evidence-based” narrative that supports any scientific view one wishes to advance, however erroneous, and be it on climate change, alternative medicine, nutrition, vaccination, evolution etc.

While it will never be possible to stop people from creating narratives to support their biases, it is essential to inform the public that peer-review is but a single criterion with which to assess the value of a publication.  We also need to remember that conclusions from even the best peer-reviewed papers should be treated with extreme caution until they are supported in high-quality papers by multiple independent lines of evidence.

So what to believe?  This is the role of web sites such as this one, and efforts like the Cochrane Collaboration, in which experts work to sort the wheat from the chafe in the scientific literature, examine the body of evidence and present their conclusions to the public in an unbiased way.

Unfortunately, this process does introduce “expert opinion” between the public and the data, but with more than 9/10 peer-reviewed publications either meaningless or full of bunk, and charlatans willing to exploit this fact to their own ends, I can’t think of an alternative.