Intended for healthcare professionals

Editorials

Clinical trials: what a waste

BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g7089 (Published 10 December 2014) Cite this as: BMJ 2014;349:g7089
  1. John P A Ioannidis, professor and director1, co-director2
  1. 1Stanford Prevention Research Center, 1265 Welch Road, Medical School Office Building, Room X306, Stanford, CA 94305, USA
  2. 2Meta-Research Innovation Center at Stanford (METRICS),
  1. Correspondence to: J P A Ioannidis jioannid{at}stanford.edu

Trials that are unregistered, unfinished, unpublished, unreachable, or simply irrelevant

Randomized controlled trials are the gold standard tool for evaluating interventions. Nevertheless, the utility of this excellent tool is contingent on how it is used. Chapman and colleagues (doi:10.1136/bmj.g6870) show this in a sample of 395 trials relevant to surgical practice that were registered in ClincialTrials.gov between 2008 and 2009.1 By the end of 2013, 21% were discontinued, 34% of those that were completed were not published, and for 77% of the trials that had uncertain fate no way existed to reach investigators to find what had happened to them. This work adds to several other empirical evaluations showing that evidence from randomized controlled trials is wasted at multiple stages from conception to publication and beyond.2 3 4 5 6 7 8 9 10 11 12

Many trials are entirely lost, as they are not even registered. Substantial diversity probably exists across specialties, countries, and settings. Overall, in a survey conducted in 2012, only 30% of journal editors requested or encouraged trial registration.2

Among registered trials, a sizeable fraction are never completed. In some cases, discontinuation may be the best course of action. Trials that prove to be futile should clearly be discontinued. Futility may develop after a trial has started, even with the best intentions and design—for example, if new and conclusive data emerge about the tested treatments. Moreover, the inability of a trial to recruit enough participants to get a meaningful answer to the research question can typically be known within a few months of the trial starting.3 However, for most trials that are discontinued early, this could probably have been avoided with more careful study design and upfront consideration of the recruitment landscape before starting the trial.

Non-publication of completed trials remains a serious and common problem across diverse specialties, as documented in multiple empirical surveys. The 34% non-publication rate reported by Chapman et al is among the more conservative estimates from studies assessing the publication rate of registered trials.4 5 6 7 It is also conservative compared with another recent study of the non-publication rate (44%) of surgical trials approved by six ethics committees.8

Paradoxically, published trials may be the weakest link in the chain of lost and distorted evidence. Strong evidence shows that only some of the original outcomes are reported and many outcomes are manipulated during analysis and reporting,9 that reported results are inflated and inferences are made with spin towards favorable conclusions,10 and that harms of interventions get second rate coverage compared with benefits.

Ideally, the whole process of conducting trials would be more accountable. Investigators starting a clinical trial should be reachable to provide information about their research and its fate. This is particularly important for trials that remain unpublished and those whose fate is uncertain. However, as Chapman et al show,1 getting even minimal information from primary investigators is difficult. Getting answers to other, more sophisticated requests such as details about protocols, analysis plans, amendments, full results, and raw data can be an ordeal.11

Finally, many trials, registered or not, completed or not, and published or not, simply represent wasted effort because the questions they ask and the comparisons and outcomes they choose to study are clinically irrelevant. Looking at the many thousands of clinical trials launched annually, this irrelevance may be actually the biggest source of waste in randomized controlled trials, although measurement of irrelevance can be subjective. The reasons why all this waste is still acceptable are complex, but largely they reflect the consequences of the current incentive system for performing clinical research.

To corporate sponsors, trials have become an unavoidable nuisance to satisfy regulators and an indispensable marketing tool. Several public funding organizations that should have been champions in supporting important, informative clinical trials, such the US National Institutes of Health, have gradually retreated from supporting randomized controlled trials. To many clinical researchers, trials have become a way to get a generous stipend (paid in proportion to the number of participants they recruit), co-author more papers, and acquire power and visibility in their professional networks. To many journals, trials offer valuable opportunities to accrue citations, influence, and reprint orders.

The perfect randomized controlled trial does not exist. Nor can perfection be reasonably expected from a design that aims to recruit and study human beings with all their wonderful diversity and freedom of choice. Moreover, the background of clinical evidence is often rapidly evolving, and seeking the best research questions to ask can feel like walking on shifting sand. However, plenty of room exists to improve the situation. Perhaps we do not need more than 20 000 clinical trials launched each year. We may do well with substantially fewer, if carefully chosen.

Trials should be properly randomized (currently more than half are not randomized at all) and use optimal study designs. They should ask key questions that matter to patients and the public, and they should be informed by a systematic examination of previous evidence.12 Trials should be well powered and use the best comparators, with pre-registration of their design and outcomes, and, whenever possible, of the analysis plan. They should avoid overt conflicts in their funding. They should be designed and conducted by non-conflicted trialists. Their results and their raw data should be publicly available and transparent. Eventually, randomized controlled trials could be the pride of clinical investigators who collaborate in research that matters, and the best source of information on how to improve health. This is what trials were supposed to be, even if we have almost totally forgotten this over the years.

Notes

Cite this as: BMJ 2014;349:g7089

Footnotes

  • Research, doi:10.1136/bmj.g6870
  • Competing interests: I have read and understood the BMJ policy on declaration of interests and declare the following interests: none.

  • Provenance: Commissioned; not externally peer reviewed.

References

View Abstract