Article Text

PDF

Standardised criteria improve accuracy of ECG interpretation in competitive athletes: a randomised controlled trial
  1. Daniel J Exeter1,
  2. C Raina Elley1,
  3. Mark L Fulcher1,
  4. Arier C Lee2,
  5. Jonathan A Drezner3,
  6. Irfan M Asif4
  1. 1Department of General Practice and Primary Health Care, The University of Auckland, Auckland, New Zealand
  2. 2Section of Epidemiology and Biostatistics, School of Population Health, The University of Auckland, Auckland, New Zealand
  3. 3Department of Family Medicine, University of Washington, Seattle, Washington, USA
  4. 4Department of Family Medicine, University of Tennessee, Knoxville, Tennessee, USA
  1. Correspondence to Dr Daniel J Exeter, Department of General Practice and Primary Health Care, The University of Auckland Private Bag 92019 Auckland Mail Centre, Auckland 1142, New Zealand; danexeter{at}gmail.com

Abstract

Background Screening to prevent sudden cardiac death remains a contentious topic in sport and exercise medicine. The aim of this study was to assess whether the use of a standardised criteria tool improves the accuracy of ECG interpretation by physicians screening athletes.

Methods Design: Randomised control trial. Study population: General practitioners with an interest in sports medicine, sports physicians, sports medicine registrars and cardiologists from Australia and New Zealand were eligible to participate. Outcome measures: Accuracy, sensitivity, specificity and false-positive rates of screening ECG interpretation of athletes. Intervention: A two-page standardised ECG criteria tool was provided to intervention participants. Control participants undertook ‘usual’ interpretation.

Results 62 physicians, with a mean duration of practice of 16 years, were randomised to intervention and control. 10 baseline and 30 postrandomisation athlete ECGs were interpreted by the participants. Intervention participants were more likely to be correct: OR 1.72 (95% CI 1.31 to 2.27, p<0.001). Correct ECG interpretation was higher in the intervention group, 88.4% (95% CI 85.7% to 91.2%), than in the control group, 82.2% (95% CI 78.8% to 85.5%; p=0.005). Sensitivity was 95% in the intervention group and 92% in the control group (p=0.4), with specificity of 86% and 78%, respectively (p=0.006). There were 36% fewer false positives in the intervention group (p=0.006).

Conclusions ECG interpretation in athletes can be improved by using a standardised ECG criteria tool. Use of the tool results in lower false-positive rates; this may have implications for screening recommendations.

Trial Registration number: ACTRN12612000641897.

  • Cardiology prevention
  • Evaluation
  • Cardiology
  • Intervention effectiveness

Statistics from Altmetric.com

Introduction

Normal cardiac adaptations from athletic training can lead to ECG findings that may be considered pathological when seen in a non-athlete. Consequently, a criticism of the use of a ‘routine’ ECG in all athletes, rather than one only performed in individuals with symptoms or a positive family history, is that it leads to an unacceptable false-positive rate.1 ,2 This may make any ECG screening programme impractical due to the costs associated with further investigations.1

ECG changes that are normal and abnormal in trained athletes are now increasingly recognised, and criteria to help distinguish physiological changes from pathological findings have been continually refined.3–11 This culminated in the Seattle Criteria, published in 2013,11 and the development of an online training course to improve ECG interpretation in athletes.12 Prior to the development of the Seattle Criteria, the European Society of Cardiology (ESC) had published guidelines for interpreting the ECGs of athletes. On the basis of these guidelines, one of the authors of this paper, JD, developed a two-page standardised ECG criteria tool.6 ,13 A 2012 study performed by two of the authors of this paper (JD and IA) showed that clinicians across grades and specialties were able to significantly improve their ability to interpret the ECGs of athletes when using this tool.14 The participants were 60 North American clinicians who interpreted the same ECGs twice: first without the tool, then a second time with the tool.

The rationale for this follow-up study was to more closely replicate clinical practice and eliminate any learning effect that may have biased the previous study. To achieve this, this study was designed such that each clinician attempted to interpret the ECGs only once, and clinicians were randomised to either use the tool (intervention) or not (control). The study aim was to assess the effect of this tool on ECG interpretation accuracy among different physician groups.

Methods

Study design

This was an individual randomised controlled trial. The ECG interpretation exercise was created as a Powerpoint presentation and sent to clinicians by email.

Study participants

Inclusion criteria

General practitioners with an interest in sports medicine, sports physicians (medical practitioners with a specialist qualification in sports medicine), sports medicine registrars (practitioners in specialist training to become sports physicians) and cardiologists from Australia and New Zealand were eligible to participate. Physicians were recruited from the databases of three Australasian organisations and through word of mouth. Officers of Sports Medicine New Zealand (SMNZ), the Australasian College of Sports Physicians (ACSP) and the Cardiac Society of Australia and New Zealand sent emails to their membership in 2011 and 2012 offering participants the opportunity to participate.

Exclusion criteria

Practitioners who were familiar with the ECG criteria tool were excluded, as were those cardiologists with a special interest in sports cardiology. Nurses were also excluded. Physicians aware of published ECG interpretation guidelines were permitted to participate as long as they were not familiar with the specific criteria tool.

Outcome measures

Outcome measures included accuracy, sensitivity, specificity and false-positive rates of screening ECG interpretation of athletes. The ECGs used in the study were the same as those used in the previous study aforementioned .14 The 40 ECGs comprised 28 normal ECGs acquired from National Collegiate Athletic Association (NCAA) Division I college athletes and 12 abnormal ECGs from individuals with known cardiac pathology. The ECGs included in the study were classified by a panel composed of four cardiologists (including a paediatric and adult electrophysiologist and two cardiomyopathy experts) and three sports medicine physicians, all experienced in the interpretation of ECGs in athletes. The normal ECGs demonstrated common ECG changes consistent with physiological adaptations of training such as sinus bradycardia, sinus arrhythmia, early repolarisation and isolated increases in QRS voltage. The abnormal ECGs represented common causes of sudden cardiac death (SCD) in young athletes with changes consistent with hypertrophic cardiomyopathy (HCM), long QT syndrome, Wolff-Parkinson-White syndrome (WPW), arrhythmogenic right ventricular cardiomyopathy (ARVC), left ventricular non-compaction and Brugada syndrome. All identifying patient information and any computer-generated interpretations were removed from the ECGs; however, interval values and axis measurements were left in place.

These ECGs were ordered by computer-generated randomisation and then divided into two groups: 1 of 10 ECGs (7 normal and 3 abnormal) and 1 of 30 ECGs (21 normal and 9 abnormal). Following recruitment and consent, each participant was enrolled and sent the initial 10 ECGs and asked to complete them using ‘usual practice’ (which could include using any ECG interpretation guides they routinely use). The purpose of performing 10 initial ECGs was to allow comparison of the baseline interpretation ability of the intervention and control groups. Participants were asked to assume that the ECGs were from asymptomatic athletes between the ages of 14 and 35 and to classify each ECG as either ‘normal or variant—no further evaluation or testing needed’ or ‘abnormal—further evaluation and testing needed’.

Intervention and control

After completing the 10 baseline ECGs, each participant was randomised to either control or intervention using computer-generated block individual randomisation. Randomisation was stratified by practitioner type and performed by an independent researcher not involved in recruitment or baseline assessment. Following randomisation, all participants were asked to complete the remaining 30 ECGs. The control group was again asked to use ‘usual practice’, whereas the intervention group was asked to use ‘usual practice’ as well as the ECG criteria tool (the intervention). The two-page ECG criteria tool used was the same as that used in a previous study by two of the coauthors (JD and IA) and subsequently published.13 This tool is based on the 2010 ESC consensus statement for interpretation of the 12-lead ECG in athletes, but with the addition of other research and input from experts familiar with cardiac screening in athletes. Therefore, when considering the evolution of ECG criteria, the tool marginally predates the Seattle Criteria, but is a step beyond the ESC guidelines.

Sample size calculations

In Drezner et al's before–after study of this tool, 60 North American physicians each interpreted 40 ECGs.14 This study was able to demonstrate significant improvements in the proportion of correctly interpreted ECGs (p<0.0001), sensitivity (from 70% to 91%) and specificity (from 89% to 94%).14 Assuming that similar differences between intervention and control groups would be achieved, we aimed for a similar sample size (60 clinicians, each interpreting 40 ECGs).

Analysis

The binary outcome of correctly interpreted ECGs was analysed using generalised linear mixed effects model with binomial distribution and logit link function to obtain ORs and associated CIs. Physician effect was analysed as random cluster effect with repeated ECGs assessed. Data from a single cluster/physician were assumed correlated with a variance component covariance matrix. Group (intervention/control), physician type and number of correct ECG interpretations at baseline were analysed as fixed effects and ECGs as random effect in the model.

Sensitivity, specificity, false positive and correct interpretation rates were calculated, with the SEs and confidence limits adjusted for clustering. Wald χ2 testing was used to compare between intervention and control groups.

Results

Seventy-five clinicians were assessed for eligibility, with 3 not being suitable clinicians (subspecialist cardiac nurses) and 10 not completing the prerandomisation ECGs by the cut-off date. This left 62 participants for randomisation (31 intervention and 31 control). One participant in the intervention group was lost to follow-up and was not analysed (see figure 1). Baseline characteristics and ECG classification scores are included in table 1.

Table 1

Baseline characteristics and ECG classification scores

Figure 1

Consolidated standards of reporting trials (CONSORT) diagram. One subject had to be removed from the study after their answers were not received by facsimile and the subject was unable to resubmit their answers.

Compared with control, intervention participants were significantly more likely to give the correct answer, OR 1.72 (95% CI 1.31 to 2.27, p<0.001), after adjusting for clustering effect, the fixed effects of practitioner type, baseline score and ECG random effect. After adjusting for clustering effect of the primary sampling unit, the percentage of postrandomisation ECGs interpreted correctly was significantly higher in the intervention group, 88.4% (95% CI 85.7% to 91.2%), than in the control group, 82.2% (95% CI 78.8% to 85.5%). Sensitivity was 94.8% (95% CI 91.1% to 98.6%) in the intervention group and 92.5% (95% CI 88.2% to 96.8%) in the control group with specificity 85.7% (95% CI 82.0% to 89.4%) and 77.7% (95% CI 73.4% to 82.0%), respectively. In the intervention group, 14% of true normal and 10% of all ECGs were incorrectly interpreted as abnormal, as opposed to 22% and 16%, respectively, in the control group. This equated to 36% fewer false positives in the intervention group. With the exception of sensitivity, all differences between the groups were statistically significant (table 2). There was no statistically significant difference between physician types, and the effect of the intervention was the same across all physician types.

Table 2

Statistical performance of control and intervention groups

Discussion

This study was able to demonstrate that interpretation accuracy across all physician groups is superior with use of this ECG criteria tool, even without specific training in how to use the tool. While the disease prevalence was arbitrarily defined at 30% in this study, the large reduction in false-positive rates has potential cost and human resource implications for screening programmes. In this study, there were 36% fewer false positives in the intervention group, which would mean 36% fewer athletes needing costly further investigations.

Hill et al15 asked paediatric cardiologists to interpret the ECGs of athletes without the use of standardised criteria, and in the subsets where the ECGs depicted HCM and myocarditis, the cardiologists were able to recommend the appropriate follow-up testing in 85% and 88% of the cases, respectively. While they were not as accurate in identifying the specific diagnosis, the role of screening ECG is to help identify those athletes that need further testing, not necessarily to make the definitive diagnosis. Therefore, 85% of the cardiologists in Hill et al's study were able to identify which of the HCM ECGs were ‘abnormal––further testing required’. This figure is comparable with the control group of cardiologists in our study.

In the previous study of this ECG tool by Drezner et al, specificity improved from 70% to 91% and sensitivity improved from 89% to 94%.14 The results of this study are not of the same magnitude as that of the previous study, mostly due to the control group of this study performing better than the first attempts in the previous study. There are a number of potential reasons for this. First, this study used a randomised controlled trial design, so results are more likely to represent the true effect of the tool—and not include the ‘learning effect’ of repeating ECG reading inherent in before–after studies. Second, ECG interpretation in athletes has attracted increased attention since the previous study was undertaken, and it is possible that some of our physicians had chosen to review this subject. Physicians were only excluded if they were familiar with the tool itself, and were allowed to participate if they were aware of the ESC guidelines that formed its basis. Finally, this paper used an Australasian population of clinicians, and the proportion of clinicians who were sports medicine trainees and specialists or cardiologists was higher than in the previous paper (74% vs 37%).

SCD remains a contentious topic in sport and exercise medicine. Recent epidemiological data suggest that SCD may be much more frequent than previously acknowledged. Recent evidence shows that SCD occurs in 1 in 43 000 college athletes per year in the USA.16 Major sporting bodies such as the IOC and FIFA and medical organisations including the American Heart Association (AHA) and the ESC agree that preparticipation cardiovascular screening should be performed.1 ,5 ,17 ,18 However, there is no unanimous support for the addition of a ‘routine’ resting 12-lead ECG to the history and examination components of the screening evaluation. The main criticism of ‘routine’ ECG is that the number of false positives is too high.1 ,19 This leads to significant expensive further testing, and consequent issues with cost-effectiveness and healthcare resources.1 Concerns that there may be a demonstrable psychological impact on athletes awaiting further tests have thus far been shown to be unfounded.20

A ‘normal’ ECG has been shown to have sensitivity as high as 95% and a negative predictive value close to 100% for HCM.21 It also has significant utility in detecting ARVC, WPW and the ion channelopathies.22 ECG is limited with respect to detecting other causes of SCD such as congenital coronary anomalies, premature coronary atherosclerosis and aortic root dilatation. History and examination alone, however, have a low sensitivity for detecting pathology.23–25

‘Training-related’ and ‘unlikely to be training-related’ ECG changes are now better understood and have been documented, most notably in 2010 with the ESC guidelines, and then more recently with the 2013 ‘Seattle Criteria’.6 ,11 These guidelines have classified many changes previously thought to be pathological as training related, reducing the number of false-positive ECGs. Accurate ECG interpretation in athletes requires physician training and knowledge to distinguish physiological changes from potential pathological findings.

The use of standardised criteria has been shown to reduce the percentage of ECGs classified as abnormal. Weiner et al26 demonstrated a reduction in false-positive rate from 16% to 10% when 508 ECGs, previously interpreted, were then re-evaluated using the 2010 ESC criteria. The major reason for the reduction in false positives was the change in the classification of isolated QRS voltage criteria for left ventricular hypertrophy from abnormal to normal. Sharma et al27 showed a further reduction in the false-positive rate from 10% to 2% by reclassifying two findings believed to be training related (voltage criteria for left atrial enlargement and right bundle branch block).

There is a potential for further improvement as the ECG criteria tool in this study was produced prior to the publication of the Seattle Criteria. Brosnan et al demonstrated a 17% false-positive rate when two experienced cardiologists used the 2010 ESC criteria to evaluate the ECGs of 1078 Australian athletes, with the false-positive rate dropping to 4.2% when the Seattle Criteria were applied to the same ECGs.28 There was no loss in sensitivity as three athletes with cardiac pathology were still detected.28 This false-positive rate is close to the 2% demonstrated in the UK and the ‘further evaluation’ rate of 2.5% shown in a US study of over 32 000 young adults.27 ,29 A tool based on the Seattle Criteria may lead to further improvements in accuracy and false-positive rates.

While cost-effectiveness has been demonstrated in a US setting when an ECG is added to a screening programme,30 there is no clear consensus on whether screening with ECG is truly cost effective. Reducing the rate of false-positive ECGs will make any programme more affordable, acceptable and easier to implement.

Study strengths and limitations

The major strength of this study was that it was a randomised control trial. It was also adequately powered to show a significant difference between the groups. The study participants were those interested in ECG screening of athletes, and as such, probably had a higher level of competence than some clinicians. However, these are the physicians most likely to be involved in everyday screening of athletes, so the findings are likely to have good generalisability. The fact that the tool was able to improve the accuracy of ECG interpretation among already competent clinicians also strengthens the findings, as there was less room for improvement.

It may be that the effect of the tool may have been greater if education about the use of the tool had been provided in addition to the tool itself. However, it is still unclear whether clinicians would be more cautious labelling an ECG as ‘normal’ if confronted with an athlete's ECG in real life. Improvements in sensitivity and specificity using the tool may also be different in a population where the proportion of disease is lower than the 30% in this study. However, if false-positive rates are a major issue, it is still likely for a substantial drop in false-positive rates to occur in a low prevalence situation. As there may be a relationship between training hours and the degree of cardiac remodelling, as well as the degree of athletic training and ECG changes,31 ,32 use of the tool in different athlete populations may produce different results.

While this study demonstrates the value of applying standardised criteria during ECG interpretation compared with usual practice, the false-positive rate is higher than published reports using expert ECG reviewers.27–29 This implies that physicians require additional training beyond simple use of a criteria tool to improve the overall accuracy of ECG interpretation before ECG screening should be considered. The Seattle Criteria were developed with the direct purpose of improving the cardiovascular care of athletes by creating a training course for physicians to acquire a common foundation in ECG interpretation in athletes. The online course reviews ECG-detectable diseases associated with SCD in athletes, physiological ECG adaptations related to athlete's heart, ECG findings suggestive of underlying pathology and the recommended secondary evaluations of abnormal ECGs, and is freely accessible to any physician.12

Conclusion

This randomised controlled trial demonstrates that use of a standardised criteria tool enhances the accuracy of ECG assessment among a range of physician types involved in athlete screening. The concomitant reduction in false positives could translate into cost-effectiveness considerations for ECG screening. Further research and education are needed to understand the training and experience necessary to conduct ECG interpretation in athletes with the greatest accuracy.

What are the new findings?

  • A randomised controlled trial shows that the use of a standardised ECG criteria tool can significantly improve the interpretation of athletes’ ECGs across a range of physicians.

  • Improvements in interpretation accuracy due to the use of a standardised ECG criteria tool are not due to any learning effect bias that may affect a before–after study.

  • In order to approach acceptable false-positive rates, the standardised ECG criteria tool must be delivered in conjunction with physician education on ECG interpretation, and not as a standalone intervention.

How might it impact on clinical practice in the future?

  • Improvements in false-positive rates may make more clinicians consider screening with ECG for their athletes.

  • A wider range of physicians may be able to be involved in the ECG interpretation of athletes allowing more wide-reaching screening programmes to be implemented.

  • The reduction in false-positive rates may improve cost-effectiveness of screening programmes.

Acknowledgments

The authors acknowledge the physicians who acted as participants in this study.

References

View Abstract

Footnotes

  • Contributors DE, CE, MF and AL contributed to the study design; JD and IA provided the intervention tool; DE, CE and AL performed the data analysis, and DE prepared the manuscript which all authors reviewed and edited.

  • Funding A grant of NZD$1500 was obtained from the Auckland Branch of Sports Medicine New Zealand (SMNZ) to assist with administrative costs.

  • Competing interests None.

  • Ethics approval University of Auckland Human Participants Ethics Committee (no. 7733).

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement Some limited, anonymised raw data is available on request from the corresponding author.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles