Article Text
PDF
Statistics from Altmetric.com
Clinicians rely on rigorous systematic reviews to guide practice. We therefore suspect many clinicians will note the results of the 2019 systematic review and meta-analysis by Webster and Hewett, ‘What is the Evidence for and Validity of Return-to-Sport Testing after Anterior Cruciate Ligament Reconstruction Surgery? A Systematic Review and Meta-Analysis’.1 We agree that it is important to evaluate the association between return-to-sport (RTS) test batteries and outcomes after ACL reconstruction. The third review question in Webster and Hewett (2019) is particularly pertinent: ‘Is passing RTS test batteries associated with reduced rates of subsequent knee injury (all knee injuries and ACL injury)?’1 We are authors of several of the original data papers cited in the systematic review, and we are concerned about the study methodology and its conclusions. We highlight major problems with including two studies and present revised analyses that demonstrate the impact these studies had on the conclusions.
Methodological concerns
First, we question the validity of pooling studies with substantial clinical and methodological diversity.2 The meta-analysis combined studies where only some athletes returned to sport3 4 and studies where all, or mostly all, returned to sport;5–7 studies with skeletally immature patients7 and studies with elite athletes;6 and studies where substantially different RTS test batteries were used. Our second concern is that Webster and Hewett1 did not assess risk of bias, a fundamental precept of systematic review methodology clearly stated in the PRISMA reporting guideline.8–10 Assessment of study quality (as performed by Webster and Hewett1) does not quantify risk of bias.11 A risk of bias assessment identifies factors within studies that can skew results, and these factors must be considered carefully in the decision to pool data and in the conclusion. Important bias domains for review questions 2 and 3 include (i) study participation, (ii) study attrition, (iii) methods used to ascertain RTS pass status, (iv) outcome (subsequent injury), (v) confounding, and (vi) statistical analysis and reporting.12
How two studies designed to assess different research constructs impacted contralateral ACL injury results
We believe that Webster and Hewett’s report of 235% greater risk of contralateral ACL injury among those who passed RTS criteria1 is an artefact of including two studies that were not designed to answer the same research question as the three other studies in the meta-analysis. In the studies by Sousa et al3 and Wellsandt et al,4 the RTS test results were used to determine when athletes were cleared to return to sport. Early return to sport is among the strongest risk factors for reinjury,5 13–15 and delaying return to sport in those who initially fail RTS testing is likely to protect them from reinjury. In these two studies,3 4 reinjuries were reported irrespective of whether the patients returned to sport after RTS testing. The three other studies5–7 in the meta-analysis represented athletes who returned to sport even if they failed RTS tests (all patients returned to sport in the studies by Kyritsis et al6 and Grindem et al,5 and 39 of 42 patients returned to sport in Graziano et al7). These patients returned to sport either because of nonadherence to the protocol or because the RTS test battery was not used for sports clearance in the sample. We contend that the clinically relevant question of whether patients should pass RTS tests prior to return to sport cannot be informed by studies where return to sport was delayed if the patient failed the RTS tests3 4. Pooling these studies3 4 in a meta-analysis is therefore inappropriate.
In the study by Sousa et al, confounding may also play a large role and is clearly highlighted in the paper’s conclusion.3 Patients who passed RTS criteria in their study were younger, had higher preinjury and follow-up activity levels, and returned to sport earlier.3 Young age,16–23 high activity levels,17 18 22 and early return to sport5 13–15 are very strong risk factors for second ACL injury; we expect higher injury rates in the ‘passed RTS criteria’ group than in the older ‘failed RTS criteria’ group who participated in less knee-demanding sports and had delayed return to sport. The majority (9/16) of the contralateral ACL injuries reported in Webster and Hewett1 were derived from the Sousa et al study,3 which was the only individual study that showed a higher rate of contralateral ACL injuries in the group that passed RTS tests. This single study accounted for 77% of the weighting for the contralateral ACL injury meta-analysis and heavily influenced the conclusions drawn by Webster and Hewett.1
Four of the remaining seven contralateral ACL injuries among the athletes who passed the RTS test in the meta-analysis were derived from the study by Wellsandt et al.4 This study was designed to evaluate estimated preinjury capacity (EPIC) levels as alternatives to limb symmetry indexes, not the association between the established RTS criteria and second ACL injuries. Patients in this study4 only received RTS clearance after they passed RTS testing,24 either 6 months after ACLR (the test point reported in Wellsandt et al4) or at a later time-point after completing additional rehabilitation. Having no contralateral ACL injuries among those who ‘failed’ the RTS test battery six months after surgery was not surprising as these patients did not return to sport at that time. Instead, athletes who failed the RTS tests continued rehabilitation and were scheduled for a new test at a later time. This prevents a meaningful comparison because no one in this specific sample returned to sport until they passed RTS criteria. As authors, we acknowledge that this information is not explicitly stated in the paper. Had we known the data would be used for another purpose than the original paper, we would have clarified to avoid this misinterpretation.
Revised analysis and interpretation of results
Here we demonstrate how conclusions change when the two studies3 4 with critically different study designs are excluded from the meta-analysis.
In the meta-analysis for any knee injury, Sousa et al3 and Wellsandt et al4 were excluded by Webster and Hewett, so their analysis remains unchanged. Passing RTS test batteries is associated with 72% lower risk of further knee injury (i.e., ACL injury and other knee injuries, 95% CI: 6% to 96% lower risk, p=0.09).1
Webster and Hewett1 reported no significant association (p=0.68) between successfully passing the RTS test and having a lower rate of subsequent ACL injury (ie, graft rupture and/or contralateral ACL injury). However, after excluding the studies by Sousa et al3 and Wellsandt et al,4 those athletes who passedthe RTS criteria had75%lower odds (95% CI: 46% to 88% lower odds) of any ACL injury than those who failed (p<0.01, figure 1).
Webster and Hewett1 found that those who passed RTS criteria had a 60% lower risk for ACL graft rupture (95% CI 31% to 77%, p=0.003). By excluding the inappropriate studies,3 4 and pooling remaining data, there is78%lower odds of graft rupture in those who passedRTS criteriacompared withthose who failed (95% CI: 52% to 90% lower odds, p<0.01, figure 2).
By excluding the studies by Sousa et al and Wellsandt et al,3 4 there are only four total contralateral ACL ruptures left for analysis (among those who passed or failed RTS criteria). These numbers aretoolowto say whether or not passing RTS criteria influences risk of contralateral ACL rupture.
Reanalysis showing the risk for any second ACL injury among those who pass versus fail RTS criteria. M-H, Mantel-Haenszel; RTS, return to sport.
Reanalysis showing the risk for an ACL graft rupture among those who pass versus fail RTS criteria. M-H, Mantel-Haenszel; RTS, return to sport.
Conclusion
Our reanalysis omits two studies3 4 with designs that addressed a different research construct than the remaining three and/or have a high risk of bias. We found that compared with patients who fail RTS tests prior to return to sport, athletes who pass RTS test batteries have: (1) a lower risk of (any) knee reinjury, (2) a lower risk of any second ACL injury and (3) a lower risk of ACL graft rupture; (4) no conclusions regarding contralateral ACL injury risk can be drawn due to insufficient data.
More evidence is needed to refine RTS test batteries to provide greater certainty in their ability to facilitate successful RTS. Future meta-analyses should critically evaluate the study design of each potential contributing paper and include a risk of bias assessment. Meta-analyses should also consider the impact of sparse data bias25 and avoid the Firth penalization when events are 0, as this method can change the direction of the reported association.26 27
In future studies, authors should report and/or control for sport level and athletic exposure. Researchers should aim to rigorously evaluate: (1) which tests can help clinicians help athletes return to play successfully, (2) the optimal values for cut-off scores and (3) alternatives to limb symmetry indexes. Importantly, all studies to date are observational, and there is a need for interventional designs (eg, pragmatic trials or site randomisation). Such studies will improve clinicians’ understanding of RTS test batteries and, with appropriate implementation, should reduce secondary knee and ACL injuries.
Clinicians should not fear an increased risk of contralateral ACL injuries on the basis of the current literature, but continue to use RTS test batteries (and appropriate time-frames) to support RTS decision-making.
Acknowledgments
Thank you to biostatistician, Dr Mohammad Ali Mansournia, MD, MPH, PhD, for his critical review of our work and for his helpful suggestions.
References
Footnotes
Correction notice This article has been corrected since it was published Online First. The order of references 5-7 has been updated.
Contributors JJC and HG contributed to conception and drafting and editing the manuscript; MAR and LSM contributed to conception and critical review; all authors read and approved the final version of the manuscript.
Funding National Institutes of Child Health & Human Development (R37 HD037985).
Competing interests JJC has received grants from the Foundation for Physical Therapy Research—Promotion of Doctoral Studies (PODS) Level I and II Scholarships and the National Institutes of Child Health & Human Development (F30 HD096830). LSM has received grants from the National Institutes of Health (R37 HD037985, R01 AR048212, T32 HD007490, R44 HD068054, U54 GM104941 and P30 GM103333). MAR has received grants from the Norwegian fund for post-graduate training in Physiotherapy. MAR and HG have received grant funding from the National Institutes of Child Health & Human Development (R37 HD037985). HG is a BJSM senior associate editor, works as a sports physiotherapist, occasionally receives minor speaking fees for presentations, and holds grants from the International Olympic Committee, the Norwegian fund for post-graduate training in Physiotherapy, and the Swedish Research Council for Sport Science.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data generated or analyzed in this manuscript are included in the systematic review and meta-analysis by Webster and Hewett.1