Article Text
PDF
Abstract
Objectives This paper provides comprehensive normative data stratified by language preference and age on the components of the National Hockey League (NHL) Sport Concussion Assessment Tool 5 (SCAT5) in a multilingual sample of professional ice hockey players and compares the findings from a paper form of the NHL SCAT5 with an electronic (App) version of the tool.
Methods A total of 1924 male NHL and American Hockey League (AHL) players (ages 17–41) were assessed during preseason medical evaluations (baseline); 1881 were assessed with the NHL SCAT5 App via tablet and 43 received the paper version of the NHL Modified SCAT5.
Results No significant differences between the App and paper modes of administration emerged in a subsample of English preference players. Significant SCAT5 differences among language preference groups emerged on measures of cognitive functioning (Immediate Memory,Concentration). No language preference differences emerged on the Delayed Recall component. Using age as a continuous variable, older participants outperformed younger players on Immediate Memory, Delayed Recall and Concentration. Players wearing skates demonstrated significantly more modified Balance Error Scoring System (mBESS) total errors than barefoot players. Normative data tables for language preference groups are presented.
Conclusions Significant differences were found between English and non-English language preference groups on the components of SCAT5, which suggest that language-specific normative data, rather than aggregated normative data, are preferable when interpreting test scores. Similarly, age-specific normative data tables may provide greater precision in data interpretation. Due to clear ceiling effects on the mBESS single leg and tandem stances, players should not be tested while wearing skates.
- concussion
- diagnosis
- evaluation
- hockey
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information. All applicable deidentified group data are presented in the text, tables and supplemental tables. Individual player data are not available.
Statistics from Altmetric.com
Introduction
The Sport Concussion Assessment Tool (SCAT)1 was designed as an educational tool for the public and to provide healthcare professionals with a standardised measure for assessing sports-related concussion (SRC). The SCAT combined previously separate assessments of symptoms,2 cognitive functioning,3 balance4 5 and a neurological screen. The SCAT has undergone three revisions: SCAT2,6 SCAT3/Child SCAT37 and the SCAT58/Child SCAT5.9 The most recent revision was labelled 5 instead of 4 to align with the fifth meeting of the Concussion in Sport Group (CISG).
The SCAT5 was designed for evaluation of suspected SRC in individuals aged 13 and older. It maintains continuity with prior versions of the SCAT while addressing limitations noted in the systematic reviews10 that guided the modification of the SCAT3 to the SCAT5/Child SCAT5. Key changes include the following:
Optional ten-word lists in addition to the five-word lists for theImmediate Memory and Delayed Recall to minimise the ceiling effects found with the SCAT3 five-word lists.
The addition of six unique sets of digit strings for the Concentration subtest.
A rapid Neurological screen to evaluate for cervical injury, speech, ability to read, balance, gait, visual tracking and finger to nose coordination.
The National Hockey League (NHL) modified the SCAT5 to conform to professional hockey by adding hockey-specific Maddocks questions and NHL visible signs, removing the five-word list option and adding a choice of symptom language presentation on the Post-Concussion Symptom Scale (PCSS). The original paper version of the NHL Modified SCAT5 (NHL SCAT5) was then incorporated into an electronic format (briefly, the ‘App’), as described later.
The purpose of this paper is to provide comprehensive normative data on the components of the NHL SCAT5 in a large sample of professional ice hockey players and to compare the paper form and theApp version. Of interest was whether the addition of the ten-word list affected the normative distribution of the Immediate Memory and Delayed Recall subtests. Additionally, the systematic review of the SCAT310 recommended undertaking a structured approach to develop comprehensive norms across language groups, sports, ages and sex. This initial step in that process examines whether score differences exist among the NHL’s multilingual players and provides normative data on key subtests by language groups.
Methods
Measures
The NHL SCAT5 App was developed by the NHL with approval of the CISG. The App reproduces all components of the original SCAT5, although modifications were made, as noted earlier. Changes in the App were made to enhance the standardised administration of the instrument by audibly presenting the Immediate Memory trials at a rate of one word per second, and the Delayed Recall component was locked by a 5 min countdown timer to prevent premature administration. Similarly, the App audibly presented digit strings for the Concentration subtest at a rate of one digit per second, with progression to the next sequence controlled by the App based on performance of prior trials. The word lists comprising the Immediate Memory subtest and the six-digit sets of the Concentration subtest were presented randomly at baseline, and a ‘look back’ feature was added for postinjury examinations to minimise athletes receiving a word list or digit string to which they had been previously exposed. Given the linguistic diversity of players in professional hockey, players could choose from eight different languages (English, Swedish, French, Russian, Czech, Finnish, Slovak or German) to endorse symptoms on the PCSS. The rest of the examination was administered in English. The symptom translations and cultural adaptions were first obtained from the commonly used ImPACT11 translated PCSS symptom lists and then verified and adapted as needed by veteran players in each of the selected languages. Lastly, countdown timers were added to all subtests requiring a timed component (eg, balance measures), and progression from one subtest to the next was dependent on completion of prior subtests.
Procedures
NHL and American Hockey League (AHL) players attending 2018 preseason medical evaluations completed the App administered via a tablet computer by team athletic trainers, physicians or neuropsychologists. A small subset of players received the paper version of the NHL SCAT5, which was used to examine form equivalence.
Analytic methods
We decided a priori to limit language-specific norms to samples larger than 20 players. When analyses failed to meet assumptions for parametric statistics (t-tests, linear regression, analysis of variance (ANOVA) with follow-up Tukey tests, analysis of covariance (ANCOVA), non-parametric statistics (Spearman correlation, χ2, Kruskal-Wallis, Mann-Whitney U) were used to examine demographic factors influencing performance. Effect size measures (eta2, rs, r, 95% CI of the difference and the common language effect size (CLES), as appropriate) were examined for all analyses and used to determine the need for sample stratification. For ease of interpretation, raw score tables are presented with corresponding percentiles. Combined App and paper administration scores are presented for symptom report. App-based normative data are presented for behavioural measures. Percentiles were created with methods described by Lenhard and colleagues12 using the cNORM package in R. This semiparametric approach uses Taylor polynomials, which have been shown to be particularly useful when working with skewed data that have ceiling or floor effects. We used the Rankit procedure option from the package to prepare the sample data and validated the obtained models using established procedures,13 14 including repeated cross-validation. More information on the obtained models along with normal plots relating raw scores, age and obtained T scores can be found in online supplementary materials. We set significance at p<0.01 to reduce type I error.
Supplemental material
Results
Preliminary analyses
Baseline assessment was completed by 1924 male hockey players during the 2018 season (mean age=24.15±4.27 years). Preferred languages included English (n=1530), Swedish (n=103), French (n=78), Russian (n=65), Czech (n=59), Finnish (n=55), Slovak (n=18) and German (n=16). Approximately 50% (n=965) of the sample denied a history of concussion, with 622 (32.32%) reporting one prior concussion, 210 (10.91%) reporting two, 83 (4.31%) reporting three and 44 (2.28%) reporting four or more concussions. Fifty (2.59%) participants reported a history of learning disability, 94 (4.89%) reported a history of Attention Deficit Hyperactivity Disorder (ADHD) and 71 (3.69%) reported a history of depression.
SCAT5 NHL App versus NHL paper administration
Forty-three players received the paper and pencil versions of the NHL SCAT5, leaving 1881 participants who were administered the SCAT5 via the App. No significant age differences were found between players who were administered the paper and App versions of the NHL SCAT5 (t(1922)=1.75, p=0.08, 95% CI of the difference −2.44 to 0.14). Since all but one player taking the paper version identified English as their preferred language, we examined paper and App form differences including only English preference players. No significantApp versus paper administration differences in performance were found on Immediate Memory (t(1528)=0.025, p=0.98, 95% CI of the difference −0.97 to 0.95), Delayed Recall (t(1528)=0.16, p=0.872, 95% CI of the difference −0.51 to 0.60), Concentration (U=30 033.50, p=0.640, CLES=0.48), modified Balance Error Scoring System (mBESS) total errors (U=22 762.50, p=0.148, CLES=0.57), symptom severity (U=28 121.50, p=0.209, CLES=0.45) or total symptoms (U=28 734.00, p=0.312, CLES=0.46).
Main analyses
Post-Concussion Symptom Scale
A Kruskal-Wallis test revealed significant language differences on total symptoms (χ2(7)=24.49, p=0.001, eta2=0.009) and symptom severity (χ2(7)=21.71, p=0.003, eta2=0.008). Follow-up Mann-Whitney U tests revealed that language differences were primarily driven by fewer symptoms among Slovak-speaking players and higher symptoms among German-speaking players. Given small sample sizes in these groups (n<20), we reran the analysis excluding players who took the PCSS in German or Slovak, revealing no significant language differences on total symptoms (χ2(5)=12.40, p=0.03, eta2=0.004) and symptom severity (χ2(5)=10.38, p=0.07, eta2=0.003).
Table 1 shows the combined (App and paper forms) PCSS normative values. Although a comparison of individual symptom differences by language preference is beyond the scope of this paper, the percentage of players (full sample) endorsing individual symptoms can be found in online supplementary figure S1. Follow-up analyses using age as a continuous variable showed that older players endorsed fewer symptoms (rs=−0.07, p=0.003) and less symptom severity (rs=−0.07, p=0.004). Because age accounted for less than 1% of the variance in symptom scores, normative data were not modelled with age as a predictor. Table 2 shows the percentiles associated with symptom and total symptom raw scores.
Descriptive statistics for baseline symptom reporting (all language groups combined; n=1890)
Percentile scores associated with SCAT5 symptom severity, total symptoms and mBESS (not wearing skates) raw scores
Balance testing
Neither age (rs=0.00, p=0.958) nor preferred language (χ2=8.85, p=0.26, eta2=0.001) was significantly associated with mBESS total errors. Table 3 presents the data for the 1620 players who completed mBESS balance testing without skates. As expected, most errors occurred on the single leg stance and the fewest errors on the double leg stance, with only 12 players (0.7%) making one or more errors. Table 2 also presents the mBESS raw scores with corresponding percentile scores. Online supplementary table S1 provides additional normative data for scores on the single leg and tandem stances performed without skates.
Descriptive statistics for mBESS total scores among NHL athletes (without skates; n=1620)
Mann-Whitney U test revealed that players wearing skates demonstrated significantly more mBESS total errors than players without skates (U=6163, p<0.001, CLES=0.97). A total of 128 players completed balance testing while wearing skates, with most of these players obtaining the maximum number of errors on mBESS tandem and single leg stances. In contrast, 83% of players completed the tandem gait without error while wearing skates, and only 3% of barefoot players made errors. Online supplementary table S2 shows the normative data for tandem gait times among players wearing and not wearing skates.
Immediate Memory
The Immediate Memory subtest was completed by 1871 players. Preliminary analyses were conducted to determine whether the three different SCAT5 ten-word groups (SCAT5 lists G, H, I) are equivalent. Significant differences were found between form 1 (list G) (M=20.81±3.23, n=654), form 2 (list H) (M=22.18±3.05, n=586) and form 3 (list I) (M=20.12±3.51, n=631) (F(2, 1868)=62.41, p<0.001, eta2=0.063). Follow-up Tukey tests revealed that performance on each form was significantly different from the others (all p<0.01). Form administration subgroups were not significantly different by age (F(2, 1868)=0.23, p=0.794, eta2<0.001) or preferred language (χ2=11.42, p=0.653). Form differences were corrected by subtracting one point from form 2 and adding one point to form 3. Follow-up ANOVA with form-corrected values revealed no significant between-group form differences (F(2, 1868)=2.41, p=0.090, eta2=0.003). Table 4 presents the descriptive statistics for forms 1–3 on the Immediate Memory and Delayed Recall tasks for English and non-English language preference groups.
Descriptive statistics for immediate memory, delayed recall and concentration
ANOVA revealed significant differences across preferred language groups on Immediate Memory total score (F(7, 1863)=7.14, p<0.001, eta2=0.026). Follow-up Tukey tests revealed that this effect was driven by English speakers significantly outperforming Czech (p<0.01) and French (p<0.01) speakers. Follow-up ANOVA removing English preference speakers revealed no significant between-group differences on Immediate Memory total score (F(6, 382)=0.53, p=0.79, eta2=0.008). We next conducted a follow-up ANCOVA with corrected Immediate Memory entered as the dependent variable and word list form entered as the independent variable. English language preference and age were entered as covariates. Both age (F(1, 1868)=46.23, p<0.001, eta2=0.024) and English language preference (F(1, 1866)=37.21, p<0.001, eta2=0.020) were significantly associated with corrected Immediate Memory. List form was not significantly associated with corrected Immediate Memory (F(2, 1866)=29.01, p=0.058, eta2=0.003). Table 5 provides the age-corrected percentile ranks for English preference and non-English preference speakers. Online supplementary figures S2 and S3 show the age-corrected normative curves for English and non-English preferred subgroups representing the relationship between raw scores and corresponding T scores.
SCAT5 ImmediateMemory percentiles associated with age and English language preference
Delayed Recall
Delayed Recall was completed by 1870 participants. ANOVA revealed significant differences between performance on form 1 (M=7.03±1.78, n=654), form 2 (M=7.48±1.70, n=586) and form 3 (M=6.55±1.84, n=630) (F(2, 1867)=42.57, p<0.001, eta2=0.044). Follow-up Tukey tests revealed that performance on all three forms was significantly different from each other (all p<0.01). In contrast, no significant differences emerged based on language preference (F(7, 1862)=1.51, p=0.16, eta2=0.006). We conducted follow-up ANCOVA with Delayed Recall entered as the dependent variable and list form entered as the independent variable. English preference and age were entered as covariates. Age (F(1, 1865)=19.47, p<0.001, eta2=0.010) and list form (F(2, 1865)=43.18, p<0.001, eta2=0.044) were significantly associated with Delayed Recall. English preference was not associated with Delayed Recall (F(1, 1865)=0.003, p=0.957, eta2<0.001). Table 6 provides the age-corrected percentile ranks for each Delayed Recall form by age. Online supplementary figures S4‒S6 show the age-corrected normative curves for Delayed Recall by list forms representing the relationship between raw scores and corresponding T scores.
SCAT5 Delayed Recall percentiles associated with age and administered form
Concentration
The Concentration portion (Digits Backwards+Months in Reverse) was completed by 1880 participants. No significant differences emerged between the six alternate digit lists (F(5, 1874)=0.24, p=0.945, eta2=0.001). ANOVA revealed significant differences between preferred language groups on Concentration total score (F(7, 1872)=23.43, p<0.001, eta2=0.081). Follow-up Tukey tests revealed that this effect was driven by English speakers outperforming Czech (p<0.01), Finnish (p<0.01), French (p<0.01), Russian (p<0.01) and Swedish (p<0.01) speakers. Follow-up ANOVA removing English preference speakers revealed no significant between-group differences on Concentration total score (F(6, 391)=1.65, p=0.133, eta2=0.025). We next conducted a follow-up ANCOVA with Concentration entered as the dependent variable, English preference entered as the independent variable and age entered as a covariate. Both age (F(1, 1877)=47.55, p<0.001, eta2=0.025) and English preference status (F(1, 1877)=123.55, p<0.001, eta2=0.062) were significantly associated with Concentration. As such, separate norms were created for English and non-English preference groups. Table 7 provides the age-corrected percentile ranks for English and non-English preference speakers. Online supplementary figures S7 and S8 show the age-corrected normative curves for English and non-English preference subgroups representing the relationship between Concentration raw scores and corresponding T scores. Online supplementary figures S9 and S10 also show the normative data for Digits Backwards.
SCAT5 Concentration percentiles associated with age and English preference
Discussion
We first endeavoured to examine whether any significant differences emerged between the paper and App form of the NHL SCAT5. The data suggest that no significant differences were found in a subset of English preference players. Additional studies with larger and more culturally diverse populations will be needed to determine whether this finding generalises beyond English speakers.
One goal of this paper was to provide comprehensive normative data on the SCAT5 in a large sample of professional ice hockey players. We first analysed normative data for the full sample and then examined whether key demographic variables (ie, language, age) moderated performance on any primary SCAT5 components.
Significant differences were found across language groups on total symptoms and symptom severity, driven mainly by German/Czech groups that were then removed from the norms due to small sample size. The final sample found no differences between any of the languages. The lack of significant differences among language groups was surprising in light of our previous work15 and that of others,16 17 which found that cultural and linguistic variables played a moderating role in symptom reporting.
Significant differences also emerged between English and other language groups on Immediate Memory, with English speakers performing significantly better than Czechs, and to a lesser extent better than Finnish, Russian and Swedish speakers. It is unsurprising that English preference players performed better than other language groups on English word verbal learning and memory, which underscores the need for culturally/demographically referenced norms when evaluating diverse populations.18 An interesting finding was that after excluding English speakers from the analyses, no significant between-group differences emerged among the remaining language groups. Perhaps those players preferring a language other than English had similar acculturation challenges with the English language and this commonality moderated any underlying intergroup differences. Comprehensive analyses with larger samples of non-English preference groups, including the use of acculturation measures, are needed to better determine whether interlanguage group differences exist.
No language preference differences emerged on Delayed Recall, which is consistent with Boone et al,19 who found no difference in delayed verbal memory performance between native English speakers and individuals speaking English as a second language. In fact, to date, no study has reported any significant effects for acculturation on delayed memory performance. As suggested by Tan and Burgess,20 it is possible that recall of previously learnt information is not a culturally bound process.
The SCAT5 ten-word lists were constructed by combining two of the original SCAT five-word lists. To our knowledge, no significant form differences have been reported with the five-word lists, so the significant differences that emerged among the three ten-word lists were unexpected. The distributions of the five-word lists on the SCAT310 21 22 exhibited significant ceiling effects. It is likely that the restricted range of these five-word lists obscured the differences that emerged with the increased variability introduced by the ten-word lists. Given these form differences, we provide score adjustments to facilitate the clinical interpretation of scores when comparing across the different forms.
The Concentration component of the SCAT5 also revealed significant differences among language preference groups, with English-speaking players significantly outperforming other language groups. This is consistent with reports of poorer Digit Span performance among foreign-born individuals.19 23 24 Peviani et al 25 speculated that non-native speakers engage additional cognitive resources for digit processing beyond those involved in the retrieval process itself.
When examined with age as a continuous variable, older players endorsed fewer symptoms and less symptom severity than younger players. Although statistically significant, the effect size was very small and not clinically significant. In contrast, the cognitive assessment components revealed that older participants outperformed younger players on Immediate Memory, Delayed Recall and Concentration. The older players’ better performance on these measures may reflect continued development of cognitive functions due to maturation effects,19 26–28 procedural familiarity with cognitive tasks due to repeated testing with similar measures (eg, list learning, Digit Span tests) throughout their tenure in professional hockey, or increased acculturation and comfort levels when tested with English-language measures due to living in North America longer than the younger players. Each of these possibilities should be examined using longitudinal study designs that include empirical assessment of acculturation, as noted earlier. In order to provide greater specificity in age norms, our age data are presented for seven different age groups. The more conventional differentiation of ‘under 20’ and ‘over 20’ may also be extrapolated from the data presented in the tables.
Neither age nor preferred language was significantly associated with mBESS total errors. Importantly, players tested while wearing skates demonstrated significantly more mBESS total errors than players tested without skates. Given the restricted range found on the mBESS single leg and tandem stances, testing players on skates using these stances is of limited clinical value and may lead to erroneous conclusions of injury-induced balance disturbances. Since hockey players are typically unwilling to remove their skates during suspected injury evaluations (due to the time required to remove and relace their skates), clinicians should consider restricting their baseline and acute postinjury evaluation of players on skates to the mBESS coordinated double leg stance and the timed tandem gait test.
Lastly, although the SCAT5 is relatively new, comparisons with limited published SCAT5 data can be made. For example, in a collegiate athlete sample,29 the average total Immediate Memory score was 20.57 (SD=3.22) with an average Delayed Recall of 6.59 (SD=1.85). A sample of high school rugby players obtained a median total score on Immediate Memory of 23 for men and 21 for women, and scores of 7 in Delayed Recall and 3 in Digits Backward for the combined sample.30 A large sample of rugby union athletes achieved an average of 21.5 (SD=3.7; median=21) on Immediate Memory and 7.1 (SD=1.9; median=7) on Delayed Recall.31 All of these studies determined that the ten-word list largely eliminated the ceiling effects found on the five-word lists. Although NHL players had similar scores on the Immediate Memory and Delayed Recall scores, the form differences across the word lists in our sample as well as the differences by language preference preclude the use of an omnibus mean across the entire sample. The ability to identify these granular differences in the data and their use to improve clinical interpretation underscore the richness of this data set.
The present study provides normative data for a large sample of male professional hockey players from diverse cultural and linguistic backgrounds. This work lays al foundation for use of normative data in the clinical interpretation of SCAT5 component scores. Future work should evaluate the SCAT5 in male and female high school/collegiate athletes playing a variety of different sports and coming from a variety of different cultural and socioeconomic backgrounds.
What are the findings?
There are no differences in the paper and App modes of administration of Sports Concussion Assessment Tool 5 (SCAT5).
SCAT5 test data differ as a function of age and language preference.
Significant differences in SCAT5 Immediate Memory and Delayed Recall scores were found among the test forms; corrections for comparisons across forms are provided.
English versus non-English language preference normative data tables for the SCAT5 are presented.
The use of skates during mBESS testing is not recommended.
How might it impact on clinical practice in the future?
Language preference and age-specific normative data tables should be used when interpreting SCAT5 test data.
Coordinated double leg stance and the timed tandem gait are preferable instead of the mBESS when testing player balance on skates.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information. All applicable deidentified group data are presented in the text, tables and supplemental tables. Individual player data are not available.
Ethics statements
Ethics approval
This study has been approved by the University of Missouri-Kansas City Office of Research Compliance (FWA #00005427).
Acknowledgments
The authors wish to acknowledge the critical contributions of the NHL medical personnel (team physicians, athletic therapists/physiotherapists and neuropsychologists) in gathering all data presented herein and their key roles in implementing the NHL/NHLPA Concussion Protocol. The authors also want to thank Dr Wolfgang Lenhard for his help with the cNORM package.
References
Footnotes
Contributors RJE and JMB assumed full responsibility for all aspects of this study, including design, data acquisition, analysis and interpretation of data, drafting and editing the manuscript, and approval of the final version and submission. JT, WM, MGH, JR and PC contributed to the design of the project, data interpretation, manuscript review and editing, and final approval/responsibility for the document. WM, PC, JR and MGH assisted in designing the study, edited and critically reviewed the manuscript, and approved the final version of the manuscript. JT assisted in drafting the manuscript and in data management, edited and critically reviewed the manuscript, and approved the final version of the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests RJE is a paid consultant for the NHL and co-chair of the NHL/NHLPA Concussion Subcommittee. He is also a paid consultant for Major League Soccer and Princeton University Athletic Medicine. He has a financial interest in EyeGuide and is chair of their Scientific Advisory Board. He is currently a co-PI for a grant funded by the NFL (NFL-Long) through Boston Children’s Hospital and occasionally provides expert testimony in matters related to MTBI and sports concussion. JT is a part-time employee of the NHL. WM is Medical Director for the NHL and an employee of the NHL. MGH is a member of the NHL/NHLPA Concussion Subcommittee and a consultant to the NHLPA, for which he receives remuneration. JR is a member of the NHL/NHLPA Concussion Subcommittee and a paid medical/physician consultant to the NHLPA. PC is co-chair of the NHL/NHLPA Concussion Subcommittee and a paid consultant to the NHLPA. JMB is a part-time employee of the NHL. He receives grant funding from Genzyme and is a consultant to Med-IQ and Sporting KC.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.