Article Text
PDF
Abstract
Background Little research examines how to best identify concussed athletes. The purpose of the present study was to develop a preliminary risk decision model that uses visible signs (VS) and mechanisms of injury (MOI) to predict the likelihood of subsequent concussion diagnosis.
Methods Coders viewed and documented VS and associated MOI for all NHL games over the course of the 2013–2014 and 2014–2015 regular seasons. After coding was completed, player concussions were identified from the NHL injury surveillance system and it was determined whether players exhibiting VS were subsequently diagnosed with concussions by club medical staff as a result of the coded event.
Results Among athletes exhibiting VS, suspected loss of consciousness, motor incoordination or balance problems, being in a fight, having an initial hit from another player’s shoulder and having a secondary hit on the ice were all associated with increased risk of subsequent concussion diagnosis. In contrast, having an initial hit with a stick was associated with decreased risk of subsequent concussion diagnosis. A risk prediction model using a combination of the above VS and MOI was superior to approaches that relied on individual VS and associated MOI (sensitivity=81%, specificity=72%, positive predictive value=26%).
Conclusions Combined use of VS and MOI significantly improves a clinician’s ability to identify players who need to be evaluated for possible concussion. A preliminary concussion prediction log has been developed from these data. Pending prospective validation, the use of these methods may improve early concussion detection and evaluation.
- Concussion
- Sports
- Ice hockey
Statistics from Altmetric.com
Introduction
Concussed athletes may be at increased risk of more severe injury or prolonged symptoms if they are not promptly removed from play1–4 As a result, the consensus guidelines published by the Concussion in Sport Group recommend the immediate removal and assessment of athletes who are suspected of having sustained a concussion.5 6 However, little research has been conducted examining how to best identify athletes with suspected concussion. Historically, the identification of athletes with possible concussion has largely been dependent on the report of athletes themselves. However, relying on athlete self-report is insufficient for several reasons. First, athletes may minimise their concussion symptoms to remain in the game. Second, the immediate effects of concussion may impair self-awareness, further reducing the likelihood that athletes will report their symptoms to medical personnel. Third, athletes may not be educated on the signs and symptoms of concussions.7–10 In response to these challenges, the NHL has developed a comprehensive concussion evaluation programme, which includes a spotter system designed to identify possible concussions based on visible signs (VS). A VS can be defined as a player’s uncharacteristic and possibly pathognomonic behaviours following a direct or indirect blow to the head that might indicate he has suffered a concussion. Examples of VS of possible concussion include—but are not limited to—behaviours such as motor incoordination and/or poor balance, lying motionless on the ice, or clutching of the head. Accordingly, as part of the NHL/NHLPA Concussion Evaluation and Management Protocol,11 athletes who exhibit VS of possible concussion are removed from the game and receive a structured clinical evaluation, which includes the Sport Concussion Assessment Tool 3rd Edition (SCAT3).6 Despite being intuitively appealing, little research has tested the diagnostic utility of this approach. Notable exceptions include recent work in rugby that concluded the use of real-time video may improve concussion management in sports.12–15
Echemendia and colleagues16 confirmed that VS of concussion can be reliably coded and used to classify concussion risk for NHL athletes. However, 53% of players diagnosed with concussions in this study exhibited no VS. Moreover, many of the VS were associated with a high number of false positives (triggering the requirement under the protocol for comprehensive concussion evaluation when there was a low probability of subsequent concussion diagnosis). When a low probability of concussion diagnosis exists, mandatory removal from competition for extensive concussion evaluation (eg, use of SCAT3) may unnecessarily disrupt the flow of professional sporting events, negatively affect athlete or team performance, and create resentment among athletes while offering little or no medical benefit.
Risk decision models help healthcare professionals make empirically informed judgements that maximise clinical utility. The development of parsimonious concussion risk models would improve care by quantifying the likelihood that a player is concussed. Players at elevated risk of concussion would then be promptly referred for evaluation, reducing the risk of additional injury. The aim of the present study was to develop a preliminary risk decision model that utilises VS and mechanisms of injury (MOI) to predict subsequent concussion diagnosis among professional ice hockey players.
Methods
Procedures
Diagnosis of concussion was made by NHL team physicians following the definition of concussion set forth by the Concussion in Sport Group,6 and the diagnosis was entered into the data analyses at the end of each season, which allowed for the inclusion of any player with delayed onset of symptoms (and hence delayed diagnosis). VS and MOI were recorded by two independent coders. Coding for MOI was completed using a reliable system developed by Hutchison and colleagues.17 Coders viewed each game from the 2013–2014 and 2014–2015 NHL regular seasons. Once each game was complete, games were then reviewed via digital recordings that included slow motion replay. If the two independent coders disagreed about the occurrence of a VS, a third coder was consulted to adjudicate and form a consensus. A consensus approach was also used to document the MOI. A more detailed review of the methods and procedures employed in this study for coding VS can be found in Echemendia and colleagues.16
The data were de-identified and analysed after receiving approval from the University of Missouri, Kansas City Office of Research Compliance (FWA #00005427).
Visible signs
As detailed in our previous work (Echemendia and colleagues),16 the following VS were coded during the course of the study. All signs required the observance of a direct or indirect hit to the head.
Suspected loss of consciousness/motionless: A player who is not moving or fails to reflexively protect or brace himself while falling after contact.
Slow to get up: A player who is hit in the head and takes longer than is typical to get up to his skates.
Motor incoordination/balance problems: A player who displays slow/laboured skating or stumbles while skating.
Blank or vacant look: A player who exhibits a vacant look or abnormalities are observed in eye position.
Clutching of head after hit: A player makes a distinct and sustained motion to grab/clutch his head (including face) or helmet with one or both hands after a contact. Exclusion: A player fixing or correcting placement of his helmet following contact.
Visible facial injury in combination with any of the above: A player suffers a visible facial injury (including ears) in which blood is observed and one of the other six visible signs is present.
Mechanism
Using reliable methods outlined in Hutchison and colleagues,17 the following mechanisms of injury were coded for each consensus event.
Initial contact: the object or body part that first hit the player who experienced the coded VS. As an example, consider the situation whereby a player is struck in the torso by an opponent’s shoulder and then falls and strikes his head on the ice. In this situation the initial contact is with the opponent’s shoulder
Secondary contact: the object or body part that hits the player after the initial hit. In the aforementioned example, the secondary contact is head to ice.
Body region of contact: the body part that is hit during the initial contact. In the aforementioend example, the body region of contact is the torso.
Result of a fight: any VS that occurs as a result of a player being hit in the head during a fight
Result of a collision: any VS that occurs as a result of a player collision
Data analysis strategy
Frequency distributions of identified MOI were calculated over the course of the study. Next, multivariable binary logistic regression was conducted with VS entered as independent variables and the presence or absence of concussion diagnosis entered as the dependent variable. A second binary logistic regression was conducted with only MOI entered as the independent variables and concussion diagnosis as the dependent variable. For variables with multiple categories (eg, secondary contact that might occur with the ice, net, boards, etc), deviation-dummy coding was used to determine the main effect of the MOI variables on concussion diagnosis. Bonferonni corrected χ2 analyses were conducted among MOI variables with a significant main effect to determine the relationship between each dichotomous subcategory (eg, secondary contact with the ice or no secondary contact with the ice) and the presence of a diagnosis of concussion. Next, the sensitivity, specificity, predictive values and likelihood ratios were calculated to determine the association between statistically significant VS, MOI and concussion diagnosis. A final multivariable logistic regression with bootstrapping (1000 samples) was conducted with significant VS and MOI variables entered as the independent variables, and concussion diagnosis entered as the dependent variable. Rounded beta weights from the remaining significant variables from this bootstrapping analysis were then used to create a preliminary risk prediction model and receiver operating characteristic (ROC) curve analyses were conducted to determine the optimal cut point for predicting the presence of subsequent concussion diagnoses. An optimal cut point was determined using Youden’s Index and risk severity cut points were determined based on clinical judgement taking into account a priori concussion probabilities determined by the obtained model. Collinearity for logistic regression models was examined using the linear regression feature of SPSS with subsequent examination of the variance inflation factor (VIF <5). IBM SPSS V.22, Microsoft Excel and Medcalc.org were used for analyses. Hosmer Lemeshow testing with χ2 analysis was used to test model calibration. With the exception of the conservative Bonferroni corrected MOI subcategory analyses used to reduce the risk of type I error, significance levels were set at p<0.01.
Results
Preliminary analyses
A total of 2460 games were reviewed by each of the two independent coders. A total of 1215 VS of concussion were coded during the study period, stemming from 861 events that occurred in 735 games. Frequencies of VS are detailed in Echemendia and colleagues.16 Table 1 shows the frequencies of the recorded MOI.
Frequency distributions of mechanisms associated with visible signs for 861 events
Analyses
A total of 202 concussions were diagnosed during the course of the 2-year study; 94 of these concussions occurred in association with an observed visible sign following a direct or indirect hit to the head. Logistic regression revealed that VS were significantly associated with subsequent concussion diagnosis (overall model χ2=77.48, p<0.001). As seen in table 2, only possible loss of consciousness/motionlessness and motor incoordination/balance problems predicted unique variance with respect to a concussion diagnosis. Logistic regression also revealed a significant relationship between MOI and subsequent concussion diagnosis (overall model χ2=124.30, p<0.001). Players who demonstrated VS as a result of a fight (OR 13.43, 95% CI 2.26 to 79.73, p=0.004) were more likely to receive subsequent concussion diagnoses. Initial contact (p=0.002) and secondary contact (p<0.001) were also significantly associated with subsequent concussion diagnosis. As shown in table 3, follow-up χ2 analyses revealed that players who were initially hit in the head or upper torso with a shoulder and players whose head hit the ice as secondary contact were more likely to receive concussion diagnoses. In contrast, both initial contact with a stick and the absence of secondary contact were associated with decreased likelihood of a subsequent concussion diagnosis. Online supplementary table 1 and table 4 show the diagnostic utility of individual significant VS and MOI. VS resulting from a collision and the body region of contact were not significantly associated with subsequent concussion diagnosis (both p values were >0.1).
Supplementary file
Multivariable logistic regression with visible signs predicting the presence of a subsequent concussion diagnosis
χ2 analyses examining the relationship between mechanism of injury sublevels and the presence of a subsequent concussion diagnosis
Diagnostic utility of significant visible signs and mechanisms of injury
Concussion prediction log.
Development of a preliminary risk decision model
Logistic regression with bootstrapping was conducted with all significant VS and MOI entered as independent variables (χ2=128.27, p<0.001). Two VS and five MOI were entered into the final model. Table 5 shows that with the exception of the presence/absence of secondary contact, all entered VS and MOI accounted for unique variance in a subsequent concussion diagnosis. Hosmer Lemeshow testing with χ2 analysis was not significant (p>0.05), indicating acceptable model calibration. Using beta values taken from table 5 and rounded to the nearest half point for ease of use, we constructed a concussion prediction log (see figure 1). Based on reliable VS and MOI coding systems, this log was designed to estimate a player’s risk of concussion—providing empirical guidelines for when players should be removed from the ice and clinically evaluated. Online supplementary figure 1 shows the results of the ROC curve analysis, indicating good discrimination (area under the curve =0.82) and an optimal cut-point from this concussion prediction log of 0.5 when using Youden’s Index. As can be seen in Table 4, the use of this cut-point provides superior diagnostic sensitivity (81%) and specificity (72%) when compared with the use of individual VS or MOI in isolation.
Logistic regression model with bootstrapping (1000 samples) predicting subsequent concussion diagnosis
Exploratory follow-up analyses
Initial contact with a stick remained in the final model but was associated with a reduced risk of subsequent concussion. We hypothesised that this may occur as a result of less severe forces associated with this type of impact. Consequently, exploratory follow-up analyses were conducted to examine whether having initial contact with a stick resulted in VS less commonly associated with concussion. Supporting the hypothesis that initial contact with a stick may be associated with less severe impact forces, players who had initial contact with a stick were significantly more likely to exhibit the VS slow to get up (96% vs 77%, χ2=41.64, p<0.001) and were also more likely to have a visible facial injury (30% vs 12%, χ2=41.40, p<0.001). In contrast, players initially hit by a stick were significantly less likely to have suspected loss of consciousness (LOC) (0% vs 3%, χ2=8.22, p<0.01).
Discussion
The development of reliable and valid methods for detecting concussion has significant implications for player health. Results of the present study demonstrate that VS used in combination with MOI might be predictive of subsequent concussion diagnoses in NHL players. Research suggests that players may be at increased risk of more severe injury or prolonged symptoms if they are not promptly removed after sustaining a concussion. In most competitive sports, athletes are evaluated for concussion if certain VS are detected (eg, LOC), or if they report symptoms of concussion. Sole reliance on self-reported symptoms in isolation is insufficient. Consequently, the NHL has instituted a policy that requires evaluation for concussion by medical personnel when certain VS are observed by medical staff or dedicated spotters. However, as demonstrated by Echemendia and colleagues16, reliance solely on VS is also problematic since 53% of the concussions in the NHL are not associated with VS and VS differ in their ability to predict a diagnosis of concussion. The results of this study indicate that when combined, both VS and MOI can add unique information when determining who should receive further evaluation for possible concussion.
The methods employed in this study can be used to inform future research on concussion identification in other professional leagues, collegiate and youth sports. Among VS, our findings indicated that suspected LOC and motor incoordination/balance problems each account for unique variance in subsequent concussion diagnoses in NHL players. In addition, we found that initial contact with the shoulder and secondary contact with the ice increase the risk of concussion diagnosis among athletes who exhibit a VS. Although concussions caused by fights are relatively uncommon in the NHL (eg, 2%), our findings indicate that players who demonstrate VS as a result of a hit to the head in a fight are at increased risk of concussion diagnosis.
In contrast to the above findings, VS that occur in the context of initial contact with a stick are associated with a decreased risk of subsequent concussion diagnosis. Prior research in this population found that less than 5% of concussions occur from direct head contact by a stick.17 Despite this, we found this MOI to be associated with a high frequency of VS. Hence, a disconnect exists between the prevalence of VS associated with initial stick contact and the proportion of concussions diagnosed. We hypothesise that non-injurious factors such as instinctive behaviour (eg, falling to the ice), competitive advantage (eg, drawing a penalty) and lesser mechanistic force, may collectively contribute to this disconnect. Alternatively, players hit by a stick may suffer injuries apart from concussion that produce VS similar to those found in concussion. Our follow-up exploratory analyses were aligned with these hypotheses, as initial contact with a stick was more likely to be associated with being slow to get up and having a visible facial injury, but less likely to be associated with suspected of loss of consciousness.
We also used logistic regression with bootstrapping and ROC curve analysis to develop a preliminary concussion risk prediction model that used both VS and MOI to predict subsequent concussion diagnoses. Using a cut score of 0.5, we obtained a sensitivity of 81% and a specificity of 72% and a positive predictive value of 26%, indicating that approximately one out of every four players referred for additional evaluation at this cut point would meet the criteria for a concussion diagnosis. Of note, the combined sensitivity and specificity of this approach was superior to using any individual VS or associated MOI to predict subsequent concussion diagnosis. Despite this, 19% of players with concussions who exhibited VS would still be missed using the established cut point. As such, a hybrid model that scales the nature of concussion evaluation based on risk may offer the most pragmatic approach. Players with VS who are in the lower risk category may benefit from briefer rink-side assessment to determine whether further evaluation is needed. In contrast, players in the elevated or high-risk categories would benefit from automatic removal and standardised assessment with evaluative tools such as the SCAT3. Variants of the obtained model should be validated with prospective samples to further identify limitations and strengths.
The present study has limitations. The NHL has dedicated VS spotters and the NHL/NHLPA concussion protocol mandates that players exhibiting certain VS following a direct or indirect hit to the head shall be removed from the ice and administered a standardised concussion evaluation by medical personnel. However, it is nonetheless possible that some of the athletes exhibiting VS were not diagnosed with a concussion when they should have been or were misdiagnosed with a concussion erroneously, for any number of reasons, including lack of sensitivity in diagnostic instruments and the imprecise nature of the current definitions of concussion. Moreover, as noted in our previous work, more than 50% of players diagnosed with concussion do not exhibit any VS in the immediate postinjury period, which underscores the importance of following players who may experience delayed symptom onset. The identification and diagnosis of concussion is a complex, challenging task for which no perfect system exists. The challenge is to continually evaluate and work to improve existing systems based on additional research as it becomes available. Future research may wish to examine the VS most predictive of concussion diagnosis across diverse sports settings and samples. Our MOI findings are most relevant to NHL athletes, though our findings lay a foundation suggesting MOI may also hold promise for concussion screening in other sports. Systematic research into MOI that may contribute to concussion identification in other sports is warranted. Future research should also examine the utility of this and other concussion identification models when compared with standard care. Finally, overfitting is a significant threat to clinical risk prediction and, consequently, the obtained cut values represent our best estimates at this moment in time among NHL players; the obtained decision rules should be validated prospectively.
In conclusion, our findings indicate that the combined use of VS and MOI significantly improves the ability to identify players who need to be evaluated for possible concussion. A concussion prediction log has been developed from these data. Pending prospective validation, the use of these methods may improve early concussion detection and evaluation, potentially reducing the severity and length of concussion symptoms.
What are new findings?
Visible signs and mechanisms of injury should be used in combination to determine who receives subsequent concussion evaluation.
Some mechanisms of injury are associated with increased risk of concussion (eg, secondary hit of the head with the ice) and some are associated with a decreased risk of concussion (eg, initial hit with a stick to the head).
Though our preliminary risk prediction model is specific to professional hockey, our methods lay a foundation for applied research in diverse sports and settings.
How might findings impact clinical practice in the near future?
Clinicians can identify mechanisms of injury and visible signs of concussions to help determine which players may need to be evaluated.
Clinicians may use the derived model to help identify the best combinations of mechanisms of injury and visible signs to identify possible concussions.
Spotter programmes may be developed to assist in the early identification of concussive injuries.
Acknowledgments
The authors would like to thank the NHL team physicians and athletic trainers/physiotherapists for their continued support and cooperation. We also would like to thank the coders in this project who worked tirelessly in a very critical role.
Footnotes
Contributors JMB and RJE took primary responsibility for the design and execution of this study, including primary responsibility for writing the manuscript. JMB took primary responsibility for data analyses. WM was involved in the design of the study and contributed to the writing and critical editing of the manuscript. MGH was involved in the design of the study and contributed to the writing and critical editing of the manuscript. PC contributed to the design of the study and review of the manuscript. MA contributed to the design of the study and review of the manuscript.
Competing interests RJE is Co-Chair of the NHL/NHLPA Concussion Subcommittee. He receives financial compensation for consulting services from the NHL, Major League Soccer, US Soccer Federation, NCAA, and Princeton University. JMB is a part-time employee of the NHL who assists the NHL/NHLPA Concussion Subcommittee with program evaluation. He also works as a consultant to Sporting KC and the Princeton University Department of Athletic Medicine. WM is an employee of the NHL and a member of the NHL/NHLPA Concussion Subcommittee and Co-Chair of the NHL/NHLPA Joint Health and Safety Committee. PC is the co-chair of the NHL/PA Concussion Subcommittee and a consultant to the NHLPA, for which he receives remuneration. MH is a member of the NHL/NHLPA Concussion Subcommittee and a consultant to the NHLPA, for which he receives remuneration. MA is a member of the NHL/NHLPA Concussion Subcommittee.
Provenance and peer review Not commissioned; externally peer reviewed.