Why so many medical 'breakthroughs' turn out to be phantom findings
After years in which a handful of scientists raised the alarm about biomedical studies that cannot be independently confirmed, top US science officials are acknowledging the crisis of "irreproducibility". The problem suggests its possible much of the research literature has been compromised by widespread incompetence or downright deception.
After years in which a handful of scientists raised the alarm about biomedical studies that cannot be independently confirmed, top US science officials are acknowledging the crisis of "irreproducibility".
The problem suggests its possible much of the research literature has been compromised by widespread incompetence or downright deception.
"We have to take this seriously," Dr Francis Collins, director of the National Institutes of Health (NIH), the country's largest funder of basic biomedical research, said. He cited the waste of time and money spent trying to build on studies whose results are a mirage or whose methods are too opaque for others to follow.
In an essay in the journal , Collins warned that "the checks and balances that once ensured scientific fidelity have been hobbled" and outlined steps NIH will take to combat the "non-replication" problem.
US President Barack Obama's Council of Advisors on Science and Technology recently held a meeting on reproducibility. The council heard from researchers who have investigated the crisis and from journal editors.
Collins said high-impact journals "are looking for papers that are groundbreaking and surprising" - the kind most likely to report results that turn out to be phantoms. Pharmaceutical, biotechnology and medical diagnostic companies have become increasingly concerned about scientific papers whose claims hold the promise of leading to new medications or diagnostic tests, but that they can't reproduce.
Scientists at biotech pioneer Amgen, for instance, reported in 2011 that they could confirm only six of 53 landmark studies in cancer biology. Researchers at pharmaceutical giant Bayer announced in 2012 that only 14 of 67 attempts to confirm claims in oncology, women's health and cardiovascular disease succeeded. Officials at Novartis and AstraZeneca told a recent cancer meeting they found the same problem.
"Whether in diagnostics or pharma, you hear the same story," said Mary Lou Gantzer, chief executive officer of BioCore Diagnostics: "'We saw something interesting in a paper, and we couldn't reproduce it.'"
NIH is now encouraging journals to devote more space for scientists to detail the methodology of their experiment. That way, there is no excuse for omitting mention of a crucial step or "secret sauce".
