Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Imagine a sports injury researcher claiming: ‘the effect of the injury prevention programme we reported in the trial is unbiased because we analysed according to the intention-to-treat principle’ (ITT). This sounds appealing for clinicians, coaches and athletes. However, before implementing results from such trial, readers should consider whether the athletes in the trial actually complied with the intervention. The appealing message above from the researcher strongly depends on the ‘whereabouts’ of the athletes. Those in the intervention group(s) need to be fully compliant to draw a meaningful conclusion regarding the effect of the intervention.
In studies affected by low compliance, we believe that drawing a conclusion on the effect of an intervention may be misleading as low compliance may bias results if data are analysed according to the ITT principle.1 We encourage clinicians, coaches and athletes to take a sceptical, cautious step back when reading bombastic conclusions in the sports injury literature. Researchers should do their best to deal with low compliance during (1) study design, (2) data collection and (3) when analysing data from sports injury trials. As a part of a BJSM educational series on methods in randomised controlled trials (RCTs), we discuss here an alternative analytical approach to the ITT—‘instrumental variable (IV) analysis’.2
Low compliance is problematic
To illustrate the problem of compliance, we used a data set from an RCT with 40-week follow-up investigating the efficacy of an Athletics Injury Prevention Programme (AIPP) to reduce injury complaints leading to restrictions in athletics participation (PREVATHLE: CPP Ouest II-Angers, number: 2017-A01980-53; ClinicalTrials.gov Identifier: NCT03307434). In this trial, track and field athletes (n=840) were allocated to either a control group (n=391, reference group), who were asked to continue their usual training, or an intervention group (n=449), who were asked to perform the specifically designed AIPP twice a week. Interestingly and thought-provokingly, only 31 of the 449 (6.9%) athletes in the intervention group complied with the AIPP (ie, twice a week) throughout the 40-week follow-up. Unfortunately, this compliance problem is well known as the low compliance of 6.9% is similar to findings in some other sports injury trials.1 Consequently, the results derived from the ITT analysis will likely not reveal any effect of the preventive programme. Alternative analytical approaches should be considered.1
Do different analytical approaches give different results?
To illustrate the comparative benefits and challenges of different analytical approaches, we applied a time-to-event approach3 and analysed data from the PREVATHLE trial in three different ways: (1) by using the ITT principle,4 2) by using the as-treated (AT) principle4 and (3) by using the IV analyses principle (table 1).
The ‘classical’ ITT analysis revealed 3.1% fewer injuries among athletes in the intervention group compared with the control group, whereas the AT analysis–on the contrary—revealed that 12.9% more athletes in the intervention group (and who actually did the AIPP) sustained injuries. These contradictory findings may be explained by confounding as those athletes who followed the injury prevention programme did it for a reason (arrow from ‘confounders’ to ‘following the intervention’ in figure 1), which could also influence the outcome (arrow from ‘confounders’ to ‘injury’ in figure 1). For instance, having a history of previous injury could be one such confounder, since this can influence athletes attend to prevention strategies and thus actually do the AIPP. In addition, previous injury has also been reported as risk factor of future injury. In this specific example, 95% of the compliant athletes reported a history of injury in the previous season. Among non-compliant athletes, 65% had had previous injury. Consequently, history of having previous injury could (alongside other variables) serve as a marker for a confounder.
In contrast to ITT and AT analyses, the IV analysis revealed that 52.5% fewer athletes sustained injury among those who did the AIPP compared with those who did not. This approach clearly underpins that results from ITT and AT can suffer from bias and provide misleading results.
What can IV analysis add?
As a practical application, we believe the IV analysis can serve (1) as a tool to crosscheck the potential risk of bias in ITT or AT analyses and (2) to provide a more accurate estimate of the efficacy of the intervention in RCT with low compliance taken into account unmeasured confounding.1 3 5–7 Similar to others,1 we recommend that IV analysis should be included, at least, as a complementary analysis to ITT used for RCT analysis,2 and is also suitable for observational studies.5 6
To be considered as an IV, a variable must meet three conditions: (1) it has a causal effect on the intervention, (2) it indirectly affects the outcome through intervention and (3) does not share common cause with the outcome.5 6 The IV could thus be the assignment to an intervention (eg, an injury prevention programme).7 IV analysis allows one to estimate the effect of the intervention on sports injury even if there are unmeasured confounding of its effect.6 It, thus, allows correction for confounding (eg, non-compliance), which is of particular interest in low compliance RCTs.1 2 6 7 The directed acyclic graph in figure 1 shows how the ITT and AT analyses are vulnerable to measured and unmeasured confounding, and how the IV analysis can address using measured confounders.8 9
What are the challenges with IV analysis?
The main limitation of IV analysis is to find a good IV which meet the conditions.2 5 6 Another comes when the IV is weakly correlated with the exposure.5 6 Sample size also could be a limitation as it can make the model is questionable.2 3 5 In our example, the extremely wide (and unrealistic) 95% CI reported in table 1 illustrates the error that surface when the assumption regarding a strong correlation between the instrument and the exposure that was violated. Importantly, alternative methods, such as non-parametric bootstrapping, than the IV-method above can be used. If such methods had been used, it may have been possible to calculate appropriate confidence intervals.
We illustrated how the IV-methods could be used if compliance to the intervention was less than 100% and the number of events are high enough. In addition, we illustrated that estimates from complementary analytical approaches (intention-to-treat, as treated and IV-analysis) can be vastly different. However, we recognise that a confidence interval should not include impossible values.10 Consequently, the present dataset should not be used in combination with the used IV estimator because the compliance to the intervention was too low and, hence, the assumption of a strong correlation between the instrument and the exposure is violated. As compliance to the intervention is low in many sports injury trials, readers should be aware that the limitations of IV-analysis can outweight its usefulness. Since IV-methods are complex, we recommended that statisticians should be included from the beginning of the study wherever possible. These should have access to the dataset in order to assist the analyst in performing the analyses.
The intention-to-treat (ITT) principle analyses according to assigned intervention regardless of the participants’ compliance with the assigned intervention. ITT analysis is often considered to provide an unbiased estimate of the effect (efficacy) of an intervention (because groups are comparable). This consideration is questionable if the level of compliance to the intervention is low. Low compliance can lead to biased results in the ITT setting if the purpose is to investigate the effect of the intervention.
The as-treated principle analyses interventions actually performed without excluding any study participants to compare compliant and non-compliant participants. However, these results can suffer from bias because confounding factors may influence both compliance and the outcome of interest (eg, lower-risk subjects not expecting any benefits of the intervention and thus exhibiting lower compliance).
The instrumental variable (IV) analysis investigates interventions actually performed while using the randomised assignment as an IV to control for confounding due to low compliance. IV analysis, or other G-estimation, should be included, at least, as a complementary analysis to ITT used for randomised controlled trial analysis, and is also suitable for observational studies.
Patient consent for publication
The randomised controlled trial called PREVATHLE was reviewed and approved by the Committee for the Protection of Persons (CPP Ouest II – Angers, number: 2017-A01980-53).
The authors warmly thank the athletes who participated in the PREVATHLE study; the athletes, coaches, physiotherapists and physicians who helped develop the Athletics Injury Prevention Programme; Marie Peurrière and Laurie Sahuc who helped for the Committee for the Protection of Persons; Jean-Michel Serra and Frédéric Depiesse from the French Athletics Federation for their support of the project, and Pierre Gardet for the development of the data collection system.
Twitter @PascalEdouard42, @RUNSAFE_Rasmus
Correction notice This article has been corrected since it published Online First. The title, table 1 and the conclusion have all been updated. The competing interests and provenance and peer review statements have also been corrected in the online version only.
Contributors PE: substantial contributions to the conception and design of the project, implementation of the project, data collection, analysis and interpretation of data, drafting, writing and revising of the manuscript and final approval of the version to be published. KS: substantial contributions to the analyses and interpretation of data, writing and revising of the manuscript, and final approval of the version to be published. LN: substantial contributions to interpretation of data, writing, revising of the manuscript and final approval of the version to be published. MAM: substantial contributions to the analyses and interpretation of data, writing and revising of the manuscript, and final approval of the version to be published. RON: substantial contributions to the conception and design of the manuscript, analysis and interpretation of data, drafting, writing and revising of the manuscript, and final approval of the version to be published.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests Authors PE, KS, MAM and RON are members of the BJSM editorial board.
Provenance and peer review Not commissioned; externally peer reviewed.