You are here

Article Text

Article menu

Download PDFPDF

Review
Diagnostic accuracy of the Ottawa Ankle and Midfoot Rules: a systematic review with meta-analysis
  1. Paula R Beckenkamp1,2,3,
  2. Chung-Wei Christine Lin1,
  3. Petra Macaskill4,
  4. Zoe A Michaleff1,
  5. Chris G Maher1,
  6. Anne M Moseley1
  1. 1Musculoskeletal Division, The George Institute for Global Health, Sydney Medical School, The University of Sydney, Sydney, New South Wales, Australia
  2. 2School of Science and Health, Western Sydney University, Australia
  3. 3Faculty of Health Sciences, The University of Sydney, New South Wales, Australia
  4. 4School of Public Health, The University of Sydney, Sydney, New South Wales, Australia
  1. Correspondence to Paula R Beckenkamp, The George Institute for Global Health, PO Box 5389, Sydney, NSW 2001, Australia; pbeckenkamp{at}georgeinstitute.org.au

Abstract

Objective To review the diagnostic accuracy of the Ottawa Ankle and Midfoot Rules and explore if clinical features and/or methodological quality of the study influence diagnostic accuracy estimates.

Design Systematic review with meta-analysis.

Data sources MEDLINE, EMBASE, CINAHL, SPORTDiscus and Cochrane Library.

Eligibility criteria for selecting studies Primary diagnostic studies reporting the accuracy of the Rules in people with ankle and/or midfoot injury were retrieved. Diagnostic accuracy estimates, overall and for subgroups (patient’s age, profession of the assessor and setting of application), were made. Sensitivity analyses included studies with a low risk of bias and studies where all patients received radiographs.

Results 66 studies were included. Ankle and Midfoot Rules presented similar accuracies, which were homogeneous and high for sensitivity and negative likelihood ratios and poor and heterogeneous for specificity and positive likelihood ratios (mean, 95% CI pooled sensitivity of Ankle Rules: 99.4%, 97.9% to 99.8%; specificity: 35.3%, 28.8% to 42.3%). Sensitivity of the Ankle Rules was higher in adults than in children, but the profession of the assessor did not appear to influence accuracy. Specificity was higher for Midfoot than for Ankle Rules. There were not enough studies to allow comparison according to setting of application. Studies with a low risk of bias and where all patients received radiographs provided lower accuracy estimates. Specificity heterogeneity was not explained by assessor training, use of imaging in all patients and low risk of bias.

Conclusions Study features and the methodological quality influence estimates of the diagnostic accuracy of the Ottawa Ankle and Midfoot Rules.

  • Ankle
  • Fracture
  • Injury
  • Sprain

https://doi.org/10.1136/bjsports-2016-096858

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Ankle sprains and fractures are common.1 While both have similar acute presentation, these injuries need to be distinguished as they are managed differently.2–4 The reference standard for diagnosing an ankle injury as a sprain or fracture involves the use of imaging, usually plain radiographs.4 However, the routine use of plain radiographs to diagnose an ankle injury contributes significantly to high healthcare costs, increases the waiting time in busy emergency departments and exposes patients to often unnecessary radiation.5 Therefore, the routine use of imaging should be avoided.

    The Ottawa Ankle and Midfoot Rules are the most commonly used clinical prediction rules to identify patients with a low probability of ankle and midfoot injuries that do not require radiographic examination.6 The Rules state that patients should be referred for radiographic examination or other medical imaging if they have pain in the malleolar or midfoot areas (for the ankle), the base of the fifth metatarsal or navicular bones (for the midfoot) as well as either bone tenderness over the same areas or an inability to weight bear four steps immediately after injury and when admitted to the emergency department.7

    Previous systematic reviews8 ,9 have reported that the Rules have high sensitivity but unclear specificity as individual studies provide heterogeneous specificity estimates. However, there are several limitations of these existing reviews. They primarily summarise the accuracy of the Rules when applied by medical doctors in secondary care (hospital emergency departments) but the healthcare system has undergone dramatic changes over the past decade, with the management of musculoskeletal injuries shifting to primary care and first contact care being provided by other health professionals, including nurses and physiotherapists. The impact of the profession of the person applying the Rules and setting of application on Rule accuracy has not been systematically investigated. The diagnostic research field has also progressed substantially since publication of the previous review,8 witnessing improvements in the design and reporting of studies,10 ,11 enhanced methods for evaluation of risk of bias12 and innovations in statistical methods for meta-analysis of diagnostic studies.13 A consideration of these factors in an updated review of the Rules has the potential to advance previous findings and address persisting uncertainties about specificity and sources of heterogeneity.

    The aim of this review was to systematically assess the diagnostic accuracy of the Ottawa Ankle and Midfoot Rules. The impact of patient's age, profession of the assessor and setting of application, as well as risk of bias, on reported accuracy of the Rules was evaluated.

    Methods

    Data sources and searches

    The review protocol was prospectively registered on PROSPERO (registration number 42013004723). The review followed the methods recommended by the Cochrane Collaboration14 and is reported according to the STARD statement.15

    The search combined relevant keywords and Medical Subject Headings and was conducted from inception to 26 August 2014 in the following electronic databases: MEDLINE via PubMed, EMBASE via subscription to http://www.embase.com, CINAHL via EBSCO-host, SPORTDiscus via EBSCO-host and Cochrane Library via OVID (see online supplementary file 1). The WHO Trial Registry portal was also searched and citation tracking of the included studies and relevant systematic reviews was conducted. There were no language or publication restrictions.

    Study selection

    Diagnostic studies were included if they: (1) reported on a cohort of patients presenting with midfoot and/or ankle trauma and an ankle or midfoot fracture was a differential diagnosis; (2) evaluated the diagnostic performance of the Ottawa Ankle and/or Midfoot Rules; (3) confirmed the diagnosis of an ankle or midfoot fracture with an adequate reference standard (eg, plain radiographs or proxy clinical measure such as a telephone follow-up); and (4) reported results in sufficient detail to allow reconstruction of contingency tables.

    Data extraction and quality assessment

    Two independent reviewers screened titles and abstracts and then the full text of the potentially eligible studies and extracted data from the included studies. Authors were contacted when there were incomplete or missing data.

    Study quality was assessed by two independent reviewers using the second version of the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2),12 which assesses the risk of bias and concerns about the applicability of systematic reviews of test accuracy studies in four domains: Patient selection (consecutive or random sample enrolled, case–control design and inappropriate exclusions avoided); Index test (blinded interpretation of the Rules); Reference standard (correctly excluded a fracture and blinded interpretation); and Flow and timing (appropriate interval between application of the Rules and reference standard, all patients received the reference standard and were included in the analysis). All disagreements were solved by discussion with a third reviewer.

    Data synthesis and analysis

    Diagnostic 2×2 contingency tables were created for each study detailing true positives, true negatives, false positives and false negatives and used to calculate study-level disease prevalence, likelihood ratios, sensitivity, specificity and their respective 95% CIs for the Ankle and Midfoot Rules. When it was not possible to distinguish between the Ankle and Midfoot, a general Ankle/Midfoot Rule estimate was provided.

    Preliminary analyses used scatter plots of individual study estimates of sensitivity and specificity in receiver-operating characteristic space to assess visually clinical and statistical heterogeneity between the studies for each Rule. The bivariate model16 ,17 was used to compute summary estimates of sensitivity and specificity separately for the Ottawa Ankle, Midfoot and Ankle/Midfoot Rules. For studies where more than one profession applied the Rules on the same sample and one of them included a medical doctor, the accuracy provided by the doctor was used for the overall estimate in the meta-analysis.

    A within-study analysis was planned to assess the accuracy of the Rules (Ankle, Midfoot and Ankle/Midfoot) for three subgroups: adults versus children; when applied by a medical doctor versus other health professionals; and when applied at a hospital emergency room versus a community setting. Since there were not enough studies for a within-study analysis, we conducted a between-study analysis for these subgroups, for each Rule, when there were at least three studies evaluating a subgroup for that Rule. For each model, the Akaike Information Criterion was used to assess whether random effects for sensitivity (versus a fixed effect) should be included in the bivariate model. For some analyses where the number of studies was small, model convergence was only possible with a fixed effect for sensitivity. Pooled likelihood ratios derived from bivariate models were used to calculate the post-test probability of fracture for a positive (and negative) test result for each subgroup, assuming the pretest probability to be the overall mean prevalence of fracture for each subgroup.

    Two sensitivity analyses were conducted. The first looked at the accuracy of studies that were rated as low risk of bias in the four domains of QUADAS-2 and the second included studies that used medical imaging as a reference standard in all patients. Imaging is the gold standard method to diagnose a fracture, reducing a potential source of bias.

    To explore reasons for heterogeneity, a series of bivariate models were fitted for each Rule that included a binary covariate: assessor training (yes vs no or not clear), reference standard (medical imaging in all patients vs in some patients) and risk of bias (low risk of bias in all QUADAS-2 domains vs not low risk of bias in all domains) for both sensitivity and specificity. These covariates were chosen as they are known to influence accuracy estimates of diagnostic tests.18 The Akaike Information Criterion was again used to assess whether random effects for sensitivity should be included. A post hoc analysis looked at the direct comparison between the accuracy of the Ankle and Midfoot Rules using studies that evaluated both Rules, thereby reducing potential confounding. The bivariate model included a binary covariate for Rule, for both sensitivity and specificity and also allowed the variance of the random effects for specificity to differ for the two Rules. Bivariate models were fitted using Proc NLMIXED in SAS V.9.4 and figures were drawn using Review Manager V.5.3.

    Results

    Selection, characteristics and quality of studies

    The search strategy yielded 261 studies and citation tracking added another 28 studies. After duplicates were removed, 140 titles and abstracts were screened and 94 full papers and eight abstracts for which a full paper was not found were assessed for eligibility. Finally, a total of 66 studies in 68 articles (56 prospective, 9 retrospective and 3 studies which could not be classified as prospective or retrospective) were included in the review, 60 full text7 ,19–77 and 8 abstracts78–85 (figure 1). The studies provided data for 22 273 patients with 3686 ankle or midfoot fractures. The prevalence of ankle and midfoot fracture across studies ranged from 0% to 35.0%, with a mean (SD) of 16.3% (6.6). The prevalence in sports centres was 11.0%. The mean age across studies ranged from 11.0 to 47.0, with a mean (SD) of 28.3 (10.4) years. One study was excluded from the pooled analysis as no fractures were detected in the sample enrolled.72 The characteristics of the included studies are provided in online supplementary file 2.

    Figure 1
    Figure 1

    Study flow chart.

    Overall, the majority of studies presented a high risk of bias in at least one domain, with only nine studies rating low risk of bias for all domains of QUADAS-2 (see online supplementary file 3).20 ,22 ,41 ,45 ,48 ,50 ,58 ,66 ,77 For the index test domain, 58% of studies were rated as low risk of bias, 51% were low risk for patient selection and flow and timing domains and only 37% for the reference standard domain. Regarding applicability, most studies rated low for patient selection (91%), 74% for the reference standard domain and 57% for index test.

    Accuracy of the Ottawa Ankle and Midfoot Rules

    Thirty-four studies (in 33 articles) reported the accuracy of the Ankle Rules,7 ,19 ,22 ,24–27 ,29 ,32 ,35–37 ,39 ,44 ,46 ,48 ,49 ,53 ,56 ,60 ,62 ,65 ,67–69 ,74–77 ,79 ,80 ,82 ,84 15 studies (in 14 articles) reported the accuracy of the Midfoot Rules7 ,19 ,22 ,25 ,44 ,48 ,53 ,60 ,67–69 ,76 ,77 ,84 and for 31 studies it was not possible to distinguish between the ankle and midfoot, so a general Ankle/Midfoot Rules accuracy is provided20 ,21 ,23 ,28 ,30 ,31 ,33 ,38 ,40–43 ,45 ,47 ,50–52 ,54 ,55 ,57–59 ,61 ,63 ,66 ,70 ,71 ,73 ,78 ,83 ,85 (table 1 and figures 2 and 3). Pooled estimates of negative likelihood ratios and sensitivity were good, while positive likelihood ratios and specificity were poor. Study specific estimates of sensitivity were generally homogeneous (ranging from 70% to 100%); however, there was substantial heterogeneity in specificities (6–85%).

    View this table:
    Table 1

    Means (95% CI) of the pooled likelihood ratios, sensitivity and specificity and means of post-test probability (%) of a positive (+) and negative (−) result

    Figure 2
    Figure 2

    Pooled estimates (mean: filled dots, 95% confidence region: dotted circles) for Ankle, Midfoot and Ankle/Midfoot accuracies.

    Figure 3
    Figure 3

    Forrest plots for Ankle, Midfoot and Ankle/Midfoot accuracies.

    Accuracy of the Ottawa Ankle and Midfoot Rules according to subgroups

    The planned within-study comparisons looking at the accuracies by subgroup: age (adults vs children), profession of the assessor (medical doctor vs other health professionals) and setting of application (hospital emergency room vs community setting) were not possible due to the small number of studies reporting these comparisons. For the between-study comparison, the age comparison was only presented in one study,51 the profession comparison was presented in three studies (two on the Ankle Rule32 ,67 and one on the Ankle/Midfoot Rule)31 and there was no study presenting a direct comparison of setting.

    Table 1 presents summary estimates of test accuracy and fracture prevalence, by subgroup, for each Rule. The only significant difference was found in the sensitivity of the Ankle Rules according to age, where sensitivity estimates were higher in the adult population.

    Heterogeneity of specificity estimates was not explained by assessor training, use of reference standard in all patients and low risk of bias in all domains from QUADAS-2.

    Sensitivity analyses

    The first sensitivity analysis looked at studies at low risk of bias for all categories of QUADAS-2, and included only three studies for the Ankle and the Midfoot Rules22 ,48 ,77 and six studies for Ankle/Midfoot Rules.20 ,41 ,45 ,50 ,58 ,66 The second sensitivity analysis looked at studies in which the entire sample was assessed using medical imaging as the reference standard, and included 25 studies (in 24 articles) for the Ankle,7 ,19 ,22 ,24–27 ,29 ,36 ,37 ,39 ,48 ,53 ,56 ,62 ,65 ,67 ,68 ,74 ,76 ,77 ,79 ,82 ,84 12 studies (in 11 articles) for the Midfoot7 ,19 ,22 ,25 ,48 ,53 ,67 ,68 ,76 ,77 ,84 and 22 studies for the Ankle/Midfoot Rules.20 ,28 ,31 ,33 ,40 ,41 ,45 ,50–52 ,54 ,55 ,57–59 ,61 ,66 ,70 ,71 ,78 ,83 ,85 Accuracy estimates were decreased in both sensitivity analyses when compared with the overall accuracies based on all studies (table 1).

    Comparison of accuracy of Ankle and Midfoot Rules

    The comparison between Ankle and Midfoot Rules accuracies was based on 15 studies (in 14 articles)7 ,19 ,22 ,25 ,44 ,48 ,53 ,60 ,67–69 ,76 ,77 ,84 that evaluated both Rules (figure 4). Bivariate modelling of these data with a covariate for Rules (Ankle vs Midfoot) showed that allowing the variance of the random effects for specificity to differ by Rule provided the best fit (based on the Akaike Information Criterion). The variability in specificity was higher for the Midfoot Rule. There was no evidence of a difference in sensitivity between the Rules (p=0.84), but weak evidence of higher specificity for the Midfoot Rule (p=0.068).

    Figure 4
    Figure 4

    Pooled estimates (mean: filled dots, 95% confidence region: dotted circles) for Ankle and Midfoot accuracies (based on studies that evaluated both Rules).

    Discussion

    Sixty-six studies evaluating the diagnostic accuracy of the Ottawa Ankle and Midfoot Rules were included, more than double the number in the previous review.8 In contrast to the previous review, statistical pooling of sensitivity and specificity were undertaken, summary estimates of positive and negative likelihood ratios computed, and sources of heterogeneity were explored. The Rules were consistently found to have a high sensitivity and low negative likelihood ratio, indicating that a negative test result is highly informative in excluding a fracture of the ankle and midfoot and, therefore, the need for radiographic examination. Specificity and positive likelihood ratios were generally low and heterogeneous and the post hoc analysis comparing the accuracy of the Rules showed that specificity was higher for the Midfoot when compared to the Ankle Rule. Assessor training, reference standard and low risk of bias did not explain heterogeneity in specificity estimates. The sensitivity analyses showed that estimates of accuracy were slightly lower in studies with a low risk of bias for all QUADAS-2 domains and when all patients receive medical imaging as the reference standard.

    This meta-analysis provides the most contemporary and robust estimate of the diagnostic accuracy of the Ottawa Ankle and Midfoot Rules and demonstrates for the first time that the profession of the assessor does not appear to influence accuracy estimates. Strategies taken to ensure methodological rigour of this review included prospective registration and applying a sensitive search strategy. This search strategy resulted in the inclusion of double the number of primary studies compared to the previous review.8 Risk of bias of included studies was assessed using a valid and reliable tool12 and bivariate models16 ,17 were used to investigate factors influencing estimates of diagnostic accuracy. The limitations of this review are primarily related to the included studies. In particular, some prespecified analyses could not be conducted due to the small number of studies included in some categories. This also limited the strength of the conclusions on the performance of the Rules when applied by different health professionals and in different settings.

    This is the first study to evaluate objectively the impact of assessor training, type of reference standard and low risk of bias on the estimates of specificity. Our review showed that these variables did not explain the observed variability. While the limited number of studies included in some categories could be a contributing factor, it is possible that other factors (eg, blinding of the assessor of the Rules) may be responsible for this variability.

    The findings of this study suggest that application of the Rules would reduce unnecessary medical imaging by ∼30% across all settings and by 49% in sports centres (compared to imaging all patients). This indicates that a large proportion of people continue to receive medical imaging, adding costs to the healthcare system, increasing time spent in emergency departments and exposing patients to unnecessary radiation. Avoiding potentially unnecessary medical tests that may cause harm has been widely supported in the Choosing Wisely campaign86 in many countries. The creation of a secondary stepwise clinical decision tool to follow the Rules could allow a more refined selection of patients to receive imaging and perhaps increase the application of the Rules in different clinical settings.

    This is the first review to provide preliminary evidence that accuracy estimates are similar when different health professionals apply the Rules. This is clinically relevant as the results suggest that clinicians who are not medical doctors, such as nurses or physiotherapists, can apply the Ottawa Ankle and Midfoot Rules to triage patients who present with an ankle and/or midfoot injury without any reduction in the diagnostic accuracy of the Rule.

    In our review, the sensitivity of the Ankle Rules was significantly higher in adults than in children, meaning that the probability of missing an ankle fracture is higher in children than in adults. This difference could be related to the lack of bone maturity seen in children and to differences in pain perceptions between these two distinct populations. A refined version of the Rules for children, taking those factors into account, could potentially help in improving the sensitivity and perhaps specificity of the Rules in this population.

    Future research in this area should focus on two key areas. First, a robust study is required to evaluate the diagnostic accuracy of the Rules when applied in a primary care community setting, to strengthen the clinical applicability of the Rules. Second, the creation of a stepwise tool to be used in conjunction to the Ottawa Ankle and Midfoot Rules or refinement of the Rules to improve the specificity estimates would be beneficial to further reduce the rates, costs and consequences associated with our current use of medical imaging.

    This review demonstrates that estimates of the diagnostic accuracy of the Ottawa Ankle and Midfoot Rules are influenced by features of the population (higher accuracy in adults) and methodological quality of the study (higher quality studies report lower accuracy). Although the accuracy of the Rules remains unclear in community settings, it has been shown that different health professionals can apply the Rules without compromises on accuracy estimates.

    What are the findings?

    • This review demonstrates that the Ottawa Ankle and Midfoot Rules are able to correctly exclude fractures in most patients; however, as the Rules dictate imaging all who test positive, the low specificity and low prevalence of fracture means that many people without fracture will still undergo imaging and exposure to radiation.

    • Different health professionals can apply the Rules without compromises on accuracy.

    • Accuracy of the Rules remains unclear in community settings and requires further investigation.

    • Assessor training, use of medical imaging as the reference standard and low risk of bias did not explain specificity heterogeneity.

    What is already known?

    • Previous reviews support the use of the Ottawa Ankle and Midfoot Rules applied by medical doctors; however, in emergency departments, other health professionals, for example, nurses and physiotherapists, are now also responsible for the triage of patients and application of the Rules.

    • Accuracy of the Rules in community settings and when applied by different health professionals is unclear.

    • Variability in specificity has been previously reported, but its source has not yet been investigated.

    References

    1. ↵
      1. Fong DTP,
      2. Hong YL,
      3. Chan LK, et al
      . A systematic review on ankle injury and ankle sprain in sports. Sports Med 2007;37:73–94.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Hulsker CCC,
      2. Kleinveld S,
      3. Zonnenberg CBL, et al
      . Evidence-based treatment of open ankle fractures. Arch Orthop Trauma Surg 2011;131:1545–53. doi:10.1007/s00402-011-1349-7
      OpenUrlCrossRefPubMed
    3. ↵
      1. Kerkhoffs GM,
      2. van den Bekerom M,
      3. Elders LAM, et al
      . Diagnosis, treatment and prevention of ankle sprains: an evidence-based clinical guideline. Br J Sports Med 2012;46:854–60. doi:10.1136/bjsports-2011-090490
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. Matharu G,
      2. Najran P,
      3. Porter K
      . Soft-tissue ankle injuries. Trauma 2010;12:105–15.
      OpenUrlCrossRef
    5. ↵
      1. Berrington de Gonzalez A,
      2. Darby S
      . Risk of cancer from diagnostic x-rays: estimates for the UK and 14 other countries. Lancet 2004;363:345–51. doi:10.1016/S0140-6736(04)15433-0
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Tiemstra JD
      . Update on acute ankle sprains. Am Fam Physician 2012;85:1170–6.
      OpenUrlPubMed
    7. ↵
      1. Stiell IG,
      2. Greenberg GH,
      3. McKnight RD, et al
      . Decision rules for the use of radiography in acute ankle injuries: refinement and prospective validation. JAMA 1993;269:1127–32.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Bachmann LM,
      2. Kolb E,
      3. Koller MT, et al
      . Accuracy of Ottawa Ankle Rules to exclude fractures of the ankle and mid-foot: systematic review. BMJ 2003;326:417–19. doi:10.1136/bmj.326.7386.417
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Dowling S,
      2. Spooner CH,
      3. Liang Y, et al
      . Accuracy of Ottawa Ankle Rules to exclude fractures of the ankle and midfoot in children: a meta-analysis. Acad Emerg Med 2009;16:277–87. doi:10.1111/j.1553-2712.2008.00333.x
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Bossuyt PM,
      2. Reitsma JB,
      3. Bruns DE, et al
      . STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology 2015;277:826–32. doi:10.1148/radiol.2015151516
      OpenUrlCrossRefPubMed
    11. ↵
      1. Korevaar DA,
      2. Wang J,
      3. van Enst WA, et al
      . Reporting diagnostic accuracy studies: some improvements after 10 years of STARD. Radiology 2015;274:781–9. doi:10.1148/radiol.14141160
      OpenUrlCrossRefPubMed
    12. ↵
      1. Whiting PF,
      2. Rutjes AW,
      3. Westwood ME, et al
      . QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155:529–36. doi:10.7326/0003-4819-155-8-201110180-00009
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Macaskill P,
      2. Glasziou P,
      3. Irwig L
      . Meta-analysis of diagnostic tests. Wiley StatsRef: Statistics Reference Online: John Wiley & Sons, Ltd, 2014.
    14. ↵
      1. Deeks J,
      2. Wisniewski S,
      3. Davenport C
      . Chapter 4: guide to the contents of a Cochrane diagnostic test accuracy protocol 2013. In: Cochrane handbook for systematic reviews of diagnostic test accuracy version 100 [Internet]. The Cochrane Collaboration. http://srdta.cochrane.org/
    15. ↵
      1. Bossuyt PM,
      2. Reitsma JB,
      3. Bruns DE, et al
      . Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Clin Chem 2004;21:4–10.
      OpenUrl
    16. ↵
      1. Macaskill P,
      2. Gatsonis C,
      3. Deeks J, et al
      . Chapter 10: analysing and presenting results. In: Cochrane handbook for systematic reviews of diagnostic test accuracy version 100 [Internet]. The Cochrane Collaboration, 2010. http://srdta.cochrane.org/
    17. ↵
      1. Reitsma JB,
      2. Glas AS,
      3. Rutjes AW, et al
      . Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005;58:982–90. doi:10.1016/j.jclinepi.2005.02.022
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Reitsma J,
      2. Rutjes A,
      3. Whiting P, et al
      . Chapter 9: assessing methodological quality. In: Cochrane handbook for systematic reviews of diagnostic test accuracy version 100 [Internet]. The Cochrane Collaboration, 2009. http://srdta.cochrane.org
    19. ↵
      1. Aginaga Badiola JR,
      2. Ventura Huarte I,
      3. Tejera Torroja E, et al
      . Validation of the Ottawa Ankle Rules for the efficient utilization of radiographies in acute lesions of the ankle. Aten Primaria 1999;24:203–8.
      OpenUrlPubMed
    20. ↵
      1. Al Omar MZ,
      2. Baldwin GA
      . Reappraisal of use of x-rays in childhood ankle and midfoot injuries. Emerg Radiol 2002;9:88–92. doi:10.1007/s10140-002-0207-x
      OpenUrlPubMed
    21. ↵
      1. Allerston J,
      2. Justham D
      . Nurse practitioners and the Ottawa Ankle Rules: comparisons with medical staff in requesting x-rays for ankle injured patients. Accid Emerg Nurs 2000;8:110–5. doi:10.1054/aaen.2000.0103
      OpenUrlCrossRefPubMed
    22. ↵
      1. Auleley GR,
      2. Kerboull L,
      3. Durieux P, et al
      . Validation of the Ottawa Ankle Rules in France: a study in the surgical emergency department of a teaching hospital. Ann Emerg Med 1998;32:14–8.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Blackham JE,
      2. Claridge T,
      3. Benger JR
      . Can patients apply the Ottawa Ankle Rules to themselves? Emerg Med J 2008;25:750–1. doi:10.1136/emj.2008.057877
      OpenUrlAbstract/FREE Full Text
    24. ↵
      1. Boutis K,
      2. Komar L,
      3. Jaramillo D, et al
      . Sensitivity of a clinical examination to predict need for radiography in children with ankle injuries: a prospective study. Lancet 2001;358:2118–21. doi:10.1016/S0140-6736(01)07218-X
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Broomhead A,
      2. Stuart P
      . Validation of the Ottawa Ankle Rules in Australia. Emerg Med (Freemantle) 2003;15:126–32.
      OpenUrl
    26. ↵
      1. Can U,
      2. Ruckert R,
      3. Held U, et al
      . Safety and efficiency of the Ottawa Ankle Rule in a Swiss population with ankle sprains. Swiss Med Wkly 2008;138:292–6. doi:2008/19/smw-12061
      OpenUrlPubMed
    27. ↵
      1. Chande VT
      . Decision rules for roentgenography of children with acute ankle injuries. Arch Pediatr Adolesc Med 1995;149:255–8.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Chandra A,
      2. Schafmayer A
      . Clinical decision rules to rule out fractures after acute blunt ankle injuries: prospective study in Germany for the evaluation of the Ottawa Ankle Rules. Unfallchirurg 2001;104:617–21.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Clark KD,
      2. Tanner S
      . Evaluation of the Ottawa Ankle Rules in children. Pediatr Emerg Care 2003;19:73–8.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Cuello-Garcia C,
      2. Ruiz-Flores A,
      3. Ramos-Gomez L, et al
      . The Ottawa Ankle Rules: evaluation in a pediatric emergency setting in Mexico. An Pediatr (Barc) 2004;60:454–8.
      OpenUrlPubMed
    31. ↵
      1. Derksen RJ,
      2. Bakker FC,
      3. Geervliet PC, et al
      . Diagnostic accuracy and reproducibility in the interpretation of Ottawa Ankle and Foot Rules by specialized emergency nurses. Am J Emerg Med 2005;23:725–9. doi:10.1016/j.ajem.2005.02.054
      OpenUrlCrossRefPubMed
    32. ↵
      1. Fiesseler F,
      2. Szucs P,
      3. Kec R, et al
      . Can nurses appropriately interpret the Ottawa Ankle Rule? Am J Emerg Med 2004;22:145–8.
      OpenUrlCrossRefPubMed
    33. ↵
      1. Garcés P,
      2. Gurucharri S,
      3. Ibiricu C, et al
      . The Ottawa Ankle Guidelines: analysis of their validity as clinical decision guidelines in the indications of x-rays for ankle and/or middle-foot traumas. Aten Primaria 2001;28:129–35.
      OpenUrlPubMed
    34. ↵
      1. Glas AS,
      2. Pijnenburg BACM,
      3. Lijmer JG, et al
      . Comparison of diagnostic decision rules and structured data collection in assessment of acute ankle injury. CMAJ 2002;166:727–33.
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Gravel J,
      2. Hedrei P,
      3. Grimard G, et al
      . Prospective validation and head-to-head comparison of 3 ankle rules in a pediatric population. Ann Emerg Med 2009;54:534–40.e1. doi:10.1016/j.annemergmed.2009.06.507
      OpenUrlCrossRefPubMed
    36. ↵
      1. Guglielmi V,
      2. Noviello MR,
      3. Guerra V, et al
      . Ankle fracture diagnosis before x-ray using Ottawa Rules. Gazz Med Ital 2003;162:121–3.
      OpenUrl
    37. ↵
      1. Hedelin H,
      2. Goksor LA,
      3. Karlsson J, et al
      . Ultrasound-assisted triage of ankle trauma can decrease the need for radiographic imaging. Am J Emerg Med 2013;31:1686–9. doi:10.1016/j.ajem.2013.09.005
      OpenUrlCrossRefPubMed
    38. ↵
      1. Karaoglu S,
      2. Kucukardali Y,
      3. Kabak S
      . Implementation of Ottawa Ankle Rules in emergency services: unnecessary radiography requests may be reduced. Erciyes Med J 2000;22:30–4.
      OpenUrl
    39. ↵
      1. Karpas A,
      2. Hennes H,
      3. Walsh-Kelly CM
      . Utilization of the Ottawa Ankle Rules by nurses in a pediatric emergency department. Acad Emerg Med 2002;9:130–3.
      OpenUrlCrossRefPubMedWeb of Science
    40. ↵
      1. Kerr L,
      2. Hutt L,
      3. Kelly AM, et al
      . Failed validation of a clinical decision rule for the use of radiography in acute ankle injury. N Z Med J 1994;107:294–5.
      OpenUrlPubMedWeb of Science
    41. ↵
      1. Keogh SP,
      2. Shafi A,
      3. Wijetunge DB
      . Comparison of Ottawa Ankle Rules and current local guidelines for use of radiography in acute ankle injuries. J R Coll Surg Edinb 1998;43:341–3.
      OpenUrlPubMedWeb of Science
    42. ↵
      1. Knudsen R,
      2. Vijdea R,
      3. Damborg F
      . Validation of the Ottawa Ankle Rules in a Danish emergency department. Dan Med Bull 2010;57:1.
      OpenUrlPubMed
    43. ↵
      1. Lau LH,
      2. Kerr D,
      3. Law I, et al
      . Nurse practitioners treating ankle and foot injuries using the Ottawa Ankle Rules: a comparative study in the emergency department. Australas Emerg Nurs J 2013;16:110–5. doi:10.1016/j.aenj.2013.05.007
      OpenUrl
    44. ↵
      1. Leddy JJ,
      2. Kesari A,
      3. Smolinski RJ
      . Implementation of the Ottawa Ankle Rule in a university sports medicine center. Med Sci Sports Exerc 2002;34:57–62.
      OpenUrlCrossRefPubMedWeb of Science
    45. ↵
      1. Leddy JJ,
      2. Smolinski RJ,
      3. Lawrence J, et al
      . Prospective evaluation of the Ottawa Ankle Rules in a university sports medicine center: with a modification to increase specificity for identifying malleolar fractures. Am J Sports Med 1998;26: 158–65.
      OpenUrlPubMedWeb of Science
    46. ↵
      1. Leisey J
      . Prospective validation of the Ottawa Ankle Rules in a deployed military population. Mil Med 2004;169:804–6.
      OpenUrlPubMed
    47. ↵
      1. Libetta C,
      2. Burke D,
      3. Brennan P, et al
      . Validation of the Ottawa Ankle Rules in children. J Accid Emerg Med 1999;16:342–4.
      OpenUrlAbstract/FREE Full Text
    48. ↵
      1. Lucchesi GM,
      2. Jackson RE,
      3. Peacock WF, et al
      . Sensitivity of the Ottawa Rules. Ann Emerg Med 1995;26:1–5.
      OpenUrlCrossRefPubMedWeb of Science
    49. ↵
      1. Mann CJ,
      2. Grant I,
      3. Guly H, et al
      . Use of the Ottawa Ankle Rules by nurse practitioners. J Accid Emerg Med 1998;15:315–16.
      OpenUrlAbstract/FREE Full Text
    50. ↵
      1. Marinelli M,
      2. Di Giulio A,
      3. Mancini M
      . Validation of the Ottawa Ankle Rules in a second-level trauma center in Italy. J Orthop Traumatol 2007;8:16–20.
      OpenUrl
    51. ↵
      1. McBride KL
      . Validation of the Ottawa Ankle Rules: experience at a community hospital. Can Fam Physician 1997;43:459–65.
      OpenUrlPubMedWeb of Science
    52. ↵
      1. Palapa Garcia LR,
      2. Regla Marquez H
      . Usefulness of Ottawa Rules for acute injuries of ankle and foot. Rev Med Inst Mex Seguro Soc 2005;43:293–8.
      OpenUrlPubMed
    53. ↵
      1. Papacostas E,
      2. Malliaropoulos N,
      3. Papadopoulos A, et al
      . Validation of Ottawa Ankle Rules protocol in Greek athletes: study in the emergency departments of a district general hospital and a sports injuries clinic. Br J Sports Med 2001;35: 445–7.
      OpenUrlAbstract/FREE Full Text
    54. ↵
      1. Parron Cambero R,
      2. Barriga Martin A,
      3. Herrera Molpeceres JA, et al
      . Validity of the Ottawa Ankle Rules as clinical decision-making criteria when requesting x-rays in ankle and/or midfoot trauma. Rev Ortop Traumatol 2006;50:283–6.
      OpenUrl
    55. ↵
      1. Parron Cambero R,
      2. Tome Bermejo F,
      3. Barriga Martin A, et al
      . Validity of the Ottawa Rules for x-ray request in ankle and midfoot injuries in paediatric patients. Eur J Orthop Surg Traumatol 2008;18:361–4.
      OpenUrl
    56. ↵
      1. Perry S,
      2. Raby N,
      3. Grant PT
      . Prospective survey to verify the Ottawa Ankle Rules. J Accid Emerg Med 1999;16:258–60.
      OpenUrlAbstract/FREE Full Text
    57. ↵
      1. Pigman EC,
      2. Klug RK,
      3. Sanford S, et al
      . Evaluation of the Ottawa Clinical Decision Rules for the use of radiography in acute ankle and midfoot injuries in the emergency department: an independent site assessment. Ann Emerg Med 1994;24:41–5.
      OpenUrlPubMedWeb of Science
    58. ↵
      1. Pijnenburg ACM,
      2. Glas AS,
      3. De Roos MAJ, et al
      . Radiography in acute ankle injuries: the Ottawa Ankle Rules versus local diagnostic decision rules. Ann Emerg Med 2002;39:599–604.
      OpenUrlCrossRefPubMedWeb of Science
    59. ↵
      1. Pires R,
      2. Pereira A,
      3. Abreu-E-Silva G, et al
      . Ottawa Ankle Rules and subjective surgeon perception to evaluate radiograph necessity following foot and ankle sprain. Ann Med Health Sci Res 2014;4:432–5. doi:10.4103/2141-9248.133473
      OpenUrl
    60. ↵
      1. Plint AC,
      2. Bulloch B,
      3. Osmond MH, et al
      . Validation of the Ottawa Ankle Rules in children with ankle injuries. Acad Emerg Med 1999;6:1005–9.
      OpenUrlPubMedWeb of Science
    61. ↵
      1. Rodrigues P,
      2. Rosa I,
      3. Campagnolo JL
      . Validation of the Ottawa Rules for the Portuguese population: a prospective study. Acta Med Port 2011;24:713–8.
      OpenUrlPubMed
    62. ↵
      1. Rosin A,
      2. Sinopoli M
      . Impact of the Ottawa Ankle Rules in a U.S. Army troop medical clinic in South Korea. Mil Med 1999;164:793–4.
      OpenUrlPubMed
    63. ↵
      1. Salt P,
      2. Clancy M
      . Implementation of the Ottawa Ankle Rules by nurses working in an accident and emergency department. J Accid Emerg Med 1997;14:363–5.
      OpenUrlAbstract/FREE Full Text
    64. ↵
      1. Santelli F,
      2. Magnanelli R,
      3. Palloto R, et al
      . Validation of the Ottawa Ankle Rules in athletes: study ub a II level trauma center. Med Sport 2008;61:373–80.
      OpenUrl
    65. ↵
      1. Singh-Ranger G,
      2. Marathias A
      . Comparison of current local practice and the Ottawa Ankle Rules to determine the need for radiography in acute ankle injury. Accid Emerg Nurs 1999;7:201–6.
      OpenUrlPubMed
    66. ↵
      1. Spanos I,
      2. Samdanis V,
      3. Chytas A, et al
      . Implementation of the Ottawa Ankle Rules by resident orthopaedic surgeons in an emergency department. Clin Res Foot Ankle 2014;2:127.
      OpenUrl
    67. ↵
      1. Springer BA,
      2. Arciero RA,
      3. Tenuta JJ, et al
      . A prospective study of modified Ottawa Ankle Rules in a military population: interobserver agreement between physical therapists and orthopaedic surgeons. Am J Sports Med 2000;28:864–8.
      OpenUrlPubMedWeb of Science
    68. ↵
      1. Stiell IG,
      2. Greenberg GH,
      3. McKnight RD, et al
      . A study to develop clinical decision rules for the use of radiography in acute ankle injuries. Ann Emerg Med 1992;21:384–90.
      OpenUrlCrossRefPubMedWeb of Science
    69. ↵
      1. Stiell IG,
      2. McKnight RD,
      3. Greenberg GH, et al
      . Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827–32.
      OpenUrlCrossRefPubMedWeb of Science
    70. ↵
      1. Szczesny G,
      2. Sypniewski M,
      3. Deszczynski J
      . Application of the Ottawa Ankle Rules in the ankle and midfoot injuries: verification of the method on the basis of own material. Pol Orthop Traumatol 1999;64:433–9.
      OpenUrl
    71. ↵
      1. Tay SY,
      2. Thoo FL,
      3. Sitoh YY, et al
      . The Ottawa Ankle Rules in Asia: validating a clinical decision rule for requesting x-rays in twisting ankle and foot injuries. J Emerg Med 1999;17:945–7.
      OpenUrlCrossRefPubMedWeb of Science
    72. ↵
      1. van der Wees PJ,
      2. Hendriks EJ,
      3. Bruls V, et al
      . Applicability of the Ottawa Ankle Rules in primary care: results from a pilot study. J Eval Clin Pract 2011;17:1246–8. doi:10.1111/j.1365-2753.2010.01479.x
      OpenUrlCrossRefPubMed
    73. ↵
      1. Verma S,
      2. Hamilton K,
      3. Hawkins HH, et al
      . Clinical application of the Ottawa Ankle Rules for the use of radiography in acute ankle injuries: an independent site assessment. AJR Am J Roentgenol 1997;169:825–7. doi:10.2214/ajr.169.3.9275906
      OpenUrlCrossRefPubMedWeb of Science
    74. ↵
      1. Wang X,
      2. Chang SM,
      3. Yu GR, et al
      . Clinical value of the Ottawa Ankle Rules for diagnosis of fractures in acute ankle injuries. PLoS ONE 2013;8:e63228. doi:10.1371/journal.pone.0063228
      OpenUrl
    75. ↵
      1. Wynn-Thomas S,
      2. Love T,
      3. McLeod D, et al
      . The Ottawa Ankle Rules for the use of diagnostic x-ray in after hours medical centres in New Zealand. N Z Med J 2002;115:U184.
      OpenUrlPubMed
    76. ↵
      1. Yazdani S,
      2. Jahandideh H,
      3. Ghofrani H
      . Validation of the Ottawa Ankle Rules in Iran: a prospective survey. BMC Emerg Med 2006;6:3. doi:10.1186/1471-227X-6-3
      OpenUrlCrossRefPubMed
    77. ↵
      1. Yuen MC,
      2. Sim SW,
      3. Lam HS, et al
      . Validation of the Ottawa Ankle Rules in a Hong Kong ED. Am J Emerg Med 2001;19:429–32. doi:10.1053/ajem.2001.24474
      OpenUrlCrossRefPubMedWeb of Science
    78. ↵
      1. Chan LM
      . Comparison of the accuracy of Bernese Ankle Rules and Ottawa Ankle Rules in ruling out ankle and midfoot fractures. Hong Kong J Emerg Med 2010;17:422.
      OpenUrl
    79. ↵
      1. Dohin B,
      2. Lubanziado D
      . Validation of the Ottawa Ankle Rules for varus ankle trauma in children: a prospective study of 160 cases. J Bone Joint Surg B 2004;86-B(Suppl 1):20.
      OpenUrlPubMed
    80. ↵
      1. Gilligan B,
      2. Burton A,
      3. Cortez P, et al
      . Application of the Ottawa Ankle Rules in children: a validation study. Acad Emerg Med 2000;7:584.
      OpenUrl
    81. ↵
      1. Marinelli M,
      2. Cecconi S,
      3. Busilacchi A, et al
      . Validation of the Ottawa Ankle Rules in a second level trauma center. J Orthop Traumatol 2011;12:S36.
      OpenUrl
    82. ↵
      1. Pacheco-Fowler V,
      2. Mankowitz S,
      3. Rhadakrishna J, et al
      . Applicability of the Ottawa Ankle Rules in an urban pediatric population. Acad Emerg Med 1999;6:452.
      OpenUrl
    83. ↵
      1. Salomone JA III.,
      2. Langford A,
      3. Patty W
      . The application of the Ottawa Ankle Rules by advance practice nurses in an emergency department fast-track system. J Accid Emerg Med 1997;4:409.
      OpenUrl
    84. ↵
      1. Solomito A,
      2. Singal B,
      3. Radack M
      . Ankle radiography in the emergency department: a prospective validation of Ottawa Ankle Rules. Acad Emerg Med 1994;1:A64.
      OpenUrl
    85. ↵
      1. Tayeb R
      . Diagnostic value of Ottawa Ankle Rules: simple guidelines with high sensitivity. Br J Sports Med 2013;47:e3.
      OpenUrlAbstract/FREE Full Text
    86. ↵
      1. Volpp KG,
      2. Loewenstein G,
      3. Asch DA
      . Choosing wisely: low-value services, utilization, and patient cost sharing. JAMA 2012;308:1635–6. doi:10.1001/jama.2012.13616
      OpenUrlCrossRefPubMedWeb of Science

    Footnotes

    • Contributors PRB, C-WCL, ZAM, CGM and AMM conceived and designed the study. PM, PRB, C-WCL, CGM and AMM analysed and interpreted the data. PRB, C-WCL, PM, ZAM, CGM and AMM contributed by drafting the article. PM provided statistical expertise. PRB, C-WCL, ZAM, CGM, AMM collected and assembled the data. PRB is guarantor. All authors participated in the revision and final approval of the manuscript.

    • Funding PRB was funded by the International Postgraduate Research Scholarship (IPRS) and Australian Postgraduate Award (APA). C-WCL and CGM have fellowships that are funded by the National Health and Medical Research Council, Australia.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.