Article Text
PDF
Abstract
Background Early identification and evaluation of concussions is critical. We examined the utility of using visible signs (VS) of concussion in predicting subsequent diagnosis of concussion in NHL players.
Methods VS of concussion were identified through video review. Coders were trained to detect and record specific visual signs while viewing videos of NHL regular season games. 2460 games were reviewed by at least two independent coders across two seasons. The reliability, sensitivity and specificity of these VS were examined.
Results VS were reliably coded with inter-rater agreement rates ranging from 73% to 98.9%. 1215 VS were identified in 861 events that occurred in 735 games. 47% of diagnosed concussions were associated with a VS but 53% of diagnosed concussions had no VS. Of the VS, only loss of consciousness, motor incoordination, and blank/vacant look had positive likelihood ratios greater than 1, indicating a positive association with concussion diagnoses. Slow to get up and clutching of the head were observed frequently but had low positive predictive values. Sensitivity decreased and specificity increased when multiple VS occurred together.
Conclusions Non-medical personnel can be trained to reliably identify events in which VS occur and to reliably identify specific VS within each of those events. VS can be useful to detect concussion early but they are not enough since more than half of physician diagnosed concussions occurred without the presence of a visual sign. The results underscore the complexity of this injury and highlight the need for comprehensive approaches to injury detection.
- Concussion
- Concussion Signs
- Video analyses
- Concussion Diagnoses
- Ice Hockey
Statistics from Altmetric.com
Introduction
There has been increased focus on the early identification of sport-related concussion (SRC) due in part to evidence that the brain may be in a heightened state of vulnerability acutely after a concussive injury.1 2 Additional trauma during this period of vulnerability may lead to greater symptom burden with longer symptom duration.3–5 Although rare, and controversial5 the prevention of Second Impact Syndrome has also been viewed as a key reason for early removal from play.6 Consequently, the early identification of concussive injury and prompt removal from play has been a focus of the management of SRC.7 There are two ways of identifying a player with a possible concussion during live play: observe the player for any visible signs (VS) of concussion on the field or ask the player if he or she is experiencing any of the symptoms of concussion. Relying on athlete self-report is problematic for several reasons: (1) athletes may minimise their concussion symptoms to remain in the game; (2) the immediate effects of concussion may impair self-awareness, further reducing the likelihood that athletes will accurately report their symptoms to medical personnel; (3) athletes may not be educated on the signs and symptoms of concussions; or (4) there is delayed symptom onset. Irrespective of the underlying reason, many athletes will not voluntarily remove themselves from play due to suspected concussion.8–10
Observing live play for possible signs of concussion has gained popularity as a method of early identification. However, this approach also faces challenges. First, there is no definitive list of what constitutes a VS of concussion. Second, since most contact/collision sports are fast paced with collisions occurring in fractions of seconds, it is difficult to follow all players in real time during play.
In March 2011, the National Hockey League (NHL) implemented a programme that focused on identifying enumerated VS of concussion following a direct or indirect hit to the head:11
Any suspected loss of consciousness (LOC)
Slow to get up
Motor incoordination/balance problems
Blank or vacant look
Disorientation
Clutching of head
Visible facial Injury in combination with any of the above
These VS were identified through examination of video records of NHL players who had sustained concussions while playing and expert consensus of the NHL’s multidisciplinary Concussion Working Group (CWG). A unique aspect of this program is that the protocol specified that any player who exhibits one or more of the enumerated VS required a mandatory off-ice examination in a distraction-free environment. The evaluation required the use of a standardised acute concussion assessment tool such as the Sport Concussion Assessment Tool-2 (SCAT2).12 In 2015 the programme was expanded to include off-ice ‘spotters’ whose role was to watch games in real time through broadcast video feeds. If an enumerated VS was identified by the spotter following a direct or indirect hit to the head, he or she contacted the medical staff through a dedicated two-way radio system to advise medical staff of their observation.
Despite increased use of ‘spotters’ and VS, particularly among professional sports leagues, there is little research evaluating the sensitivity, specificity and reliability of VS to assess players for concussion. A notable exception is recent data presented by Makdissi and Davis13 in Australian football that examined ‘video signs’ from a random sample of videos obtained over two seasons and concluded that video analysis could be a useful adjunct to sideline evaluation in determining whether a concussion has occurred.
The goal of the present study was to examine the sensitivity and specificity of VS in predicting a subsequent diagnosis of concussion by a NHL team physician. The determination of specificity and sensitivity of signs is critically important since a large number of false positives leads to unnecessary removal of players from competition while an unacceptable number of false negatives leads to the possibility of keeping vulnerable players at risk for additional injury. Although this paper seeks to determine the reliability, sensitivity and specificity of the NHL’s VS, the results of this study may help inform concussion identification in other sports.
Methods
Procedures
Video review of prior NHL concussion injuries was conducted by members of NHL Hockey Operations and the NHL’s CWG. Individuals in the Hockey Operations department are experts in the game of hockey as many are prior NHL players. The goal of the review was to identify observable indicators that appeared to be common in players who had been diagnosed with a concussion. The CWG then aggregated the observations and arrived at a consensus of the specific observable signs that constituted the VS that were subsequently incorporated into the 2011 Protocol.11 Since the responsibility of identifying potential concussions in the NHL was deemed a club-wide responsibility, a central tenet in the identification of these signs was that the signs had to be observable and identifiable by non-medically trained individuals.
Five coders knowledgeable of the game of hockey were trained by a concussion specialist clinical neuropsychologist to detect and record these VS using operational definitions (see below) and video examples that were agreed on by the NHL/NHLPA Concussion Subcommittee (formally the CWG). Training persisted until all coders reached 80% or higher inter-rater agreement on all VS. Coders were blind to the purposes of the project or whether the players sustained a concussion. Two independent coders were randomly assigned to watch each regular season NHL game (1230 games per season) from the 2013–14 and 2014–15 seasons. Games were reviewed via video after completion of games (slow motion replay was available). Coders independently watched each game to identify events with one or more VS and were not restricted as to how many times they could review events. Coders logged their observations to indicate whether one or more VS occurred. An ‘event’ was identified if one or more VS were detected by coder consensus. The number and type of VS for each event were also coded. A project manager reviewed all of the independent log entries by the coders. If the two independent coders for each game disagreed about the occurrence of an event (ie, whether a VS occurred), a third coder independently reviewed the footage to form a consensus opinion. Similarly, if the two independent loggers disagreed about the presence of a particular VS, a third independent coder acted as tie-breaker. The project manager independently reviewed a 5% random sample of games as a reliability check on the coders. Consensus meetings were held regularly throughout the project to reduce coding drift or to clarify any misunderstandings of the operational definitions.
The diagnosis of concussion was made by NHL team physicians following the definitions of concussion set for the by the Concussion In Sport Group (CISG).7
Visible signs
The following VS (and their definitions) were coded during the course of the project. Minor adjustments to these definitions (primarily clarification) were made during the course of the project to improve consistency. In each instance the prior data were reviewed to determine whether the amended definition changed any previously coded events, and those coded events were changed accordingly (table 1).
Visible signs (VS) of possible concussion (following a direct or indirect hit to the head)
Results
The data were de-identified and analysed after receiving approval from the University of Missouri-Kansas City Office of Research Compliance (FWA #00005427). IBM Statistics V.23 and Microsoft Excel were used for statistical analyses.
Reliability
The first step in assessing the reliability of the VS involved determining the extent to which coders agreed on the identification of an ‘event', irrespective of the number of VS per event, followed by assessing agreement rates on the specific VS. Reliability of event detection and VS coding was examined in the first 240 games. VS were observed in 31.7% of these games (76/240) with 89 events being identified by at least one coder. Eighty of these events were confirmed by either both coders independently or the third consensus coder and nine were excluded after review by the third coder. Among those events identified by at least one coder, acceptable coder agreement (>70%) was found for LOC/motionless (97.8%), slow to get up (73.0%), motor incoordination (92.1%), blank or vacant Look (98.9%), disorientation (98.9%), clutching of head (87.6%) and visible facial injury in combination with any VS (95.5%). We considered using the κ statistic to examine reliability; however, due to the fact that there are a near infinite number of true negatives during the course of a game (both coders agree there is no VS), the use of this common statistic was not feasible.
VS frequencies
A total of 2460 regular season games were reviewed by each of the two independent coders who identified 1215 signs stemming from 861 events that occurred in 735 games. As noted above, a third consensus coder was used whenever there was disagreement between the coders on any VS identified by at least one coder in an event. There was disagreement among coders on 181 (15%) VS. Of those, the consensus coder confirmed the existence of a sign in 74 (41%) instances and decided that a sign was not present in 107 (59%) instances.
Turning to the VS, visible facial injury (in combination with another VS) occurred 161 times, clutching of head was identified 121 times, blank or vacant look was detected 6 times, incoordination/balance problem occurred 112 times, slow to get up was noted in 790 events and LOC/lying motionless on the ice was observed 25 times. The sign ‘disorientation’ was not observed in any event during the study period and was therefore not included in subsequent analyses.
Primary analyses
VS event data were combined with NHL injury data to determine which VS were associated with diagnosed concussions. During the course of the 2-year study, 202 concussions were diagnosed during regular season games. Of these, 94 (47%±6.88) concussions were associated with a V In other words, 108 (53%) of diagnosed concussions had no VS identified by the coders. Tables 2, 3 show the frequency and diagnostic utility of each VS in relation to subsequent concussion diagnoses. The data reveal that only LOC/motionless, motor incoordination and blank/vacant look had positive likelihood ratios >1, indicating an association with subsequent concussion diagnoses. Tables 4, 5 show the frequency and diagnostic utility of having multiple signs occurring concurrently. Combinations of signs that had high PPV were also examined. As can be seen, sensitivity decreased and specificity increased with the number of VS. Sensitivity, specificity and PPV were maximised when players exhibited one or more of the three aforementioned VS that showed positive likelihood ratios with 95% CIs >1 (LOC, motor incoordination, blank/vacant look). Despite these observations, the overall sensitivity of these significant VS was low.
Frequency of visible signs (VS) among players with diagnosed concussion in the 2013–2014 and 2014–2015 regular seasons
Sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio and negative likelihood ratio for visible signs predicting subsequent concussion diagnoses
Frequency of multiple signs and combinations of signs associated with concussion diagnosis
Sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio and negative likelihood ratio for multiple signs/combinations of signs predicting subsequent concussion diagnoses
Follow-up analyses
Because being slow to get up following a hit to the head was the most frequently occurring VS with the poorest inter-rater reliability, an attempt was made to develop a more precise operational definition of the sign by subjecting the data to Receiver Operator Characteristic (ROC) curve analysis. Figure 1 shows the results of the ROC analysis examining optimal time cut points (35 s and 18 s) using Youden’s index for maximising sensitivity and specificity.
Receiver operating characteristic curve for time to get up following a hit to the head predicting subsequent concussion diagnosis (adapted from Torres et al).10 AUC=0.698; 95% CI 0.629 to 0.768.
Discussion
The purpose of this study was to examine the reliability, sensitivity and specificity of the enumerated VS of concussions as identified in the NHL Concussion Protocol. The results indicate that non-medical personnel can be trained to reliably identify events in which VS occur in professional ice hockey and to reliably identify specific signs within each of those events. The one exception was disorientation, which did not occur in any of the games that were reviewed and was subsequently not included in additional analyses and removed from the protocol. Similarly, ‘facial laceration in combination with another visible sign’ was judged to be redundant with other signs and was also dropped from the protocol.
VS were observed in only 47% of all diagnosed concussions. In other words, more than half of physician diagnosed concussions within the NHL occurred without the presence of one of the enumerated VS. Interestingly, this is the same percentage that was identified in Australian football.13 If replicated across other studies, the identification of most concussions will continue to rely, at least in part, on player self-reported symptoms. Given the known reluctance of some athletes to report concussion symptoms, systematic education on the importance of reporting concussive symptoms will play a paramount role in the successful management of this injury.
The implications of these findings are important for assessing the adequacy of using VS in early detection of concussion. Efforts should be directed towards identifying additional signs and other factors that may be helpful in detecting this injury. Such efforts may include detailed video analysis of each concussion with the goal of identifying other signs that may exist. Although not published to date, other professional sports, including the National Football League and Major League Soccer have publicly stated that they have or are developing VS reporting protocols that may be informed by the present study. The existence of these programmes as well as the data published on professional Australian football and rugby underscore the importance of early identification of possible concussions and the need for developing novel approaches to identifying these injuries. For example, Makdissi and Davis13 include ‘impact seizure’ and ‘rag doll appearance’ among their video signs. The use of these additional signs may reflect the need for unique signs based on the characteristics of individual sports.
Alternatively, it is widely recognised that concussions are complicated injuries with robust individual differences in symptom presentation, including delayed onset of symptoms. These results further underscore the complexity of this injury and highlight the need for comprehensive multimethod/multidimensional approaches to injury detection that may include video reviews, the use of trained observers, use of psychometric measures, measures of balance, ocular-motor functioning, player report, and so on. In short, observable signs can be useful but comprise only one component of what must be a multifaceted approach.
Turning to the specific signs, the three signs with the highest PPV (LOC/motionless, blank/vacant look, motor incoordination) were among the signs with the lowest frequency of occurrence. As noted earlier, PPV is a useful indicator because it reflects the likelihood that a concussion was diagnosed when a VS was present. As such, the sign LOC/motionless with a PPV of 52% indicates that a concussion would be diagnosed 52% of the time following that VS. Conversely, the two signs with the highest frequency of occurrence (slow to get up, clutching of the head) had significantly lower PPV. Indeed, slow to get up was by far the most frequently occurring sign but its PPV was only 10%. In other words, only 1 in 10 players who exhibit this sign following a direct or indirect hit to the head will be diagnosed with a concussion. ROC analyses were conducted to improve the utility of this sign, which revealed that 35 or more seconds need to elapse between a player going down to the ice and then returning upright to his skates for maximal sensitivity and specificity. Although scientifically interesting, from a practical perspective any player who is down on the ice for 35 s or more will likely be evaluated by medical staff independent of any mandatory protocol to do so. This finding illustrates that there are many reasons why players who have been hit in the head may exhibit VS that are not related to concussive injury. For example, players may have the wind knocked out of them, they may want to ‘rest’ due to fatigue, or may be in pain from a musculoskeletal injury.
In contrast, players who appear to have lost consciousness or lay motionless on the ice, exhibit motor incoordination, or exhibit a blank or vacant look, are more likely to be diagnosed with a concussion. Players who exhibit these signs following a hit to the head should undergo a comprehensive acute examination by medical staff. However, even signs with relatively high PPV do not predict a diagnosis of concussion with certainty. It is also important to remember that although physician diagnosis of concussion remains the gold standard in determining whether a concussion has occurred, the diagnosis is complex and the criteria imprecise, and thus is not infallible. As such, it is possible that some players demonstrating VS may have experienced concussion without subsequent medical diagnosis. The development and validation of a definitive diagnostic biomarker would be required to eliminate the possibility of false negatives.
For the present study, ‘events’ were identified when one or more VS were detected by coder consensus. Consequently, obtained specificity statistics are most accurately applied to the subsample of athletes exhibiting VS. It is impossible to predict the actual number of true negatives given our study design because of the infinite number of possible true negatives (eg, players who have experienced a direct or indirect hit to the head who do not exhibit VS and do not receive a concussion diagnosis). As a result, our findings likely underestimate specificity and associated metrics. For this reason, we focus on sensitivity, positive predictive value and associated metrics. One approach for future studies may be to identify any body contact that may generate traumatic forces to the head, which in this protocol has been defined as an indirect hit to the head, and using these events to calculate the true negatives. Although imperfect, such an approach may lead to more accurate estimates of sensitivity.
Of interest are the data independently reported by Makdissi and Davis.13 In their sample of professional Australian football players they also found that blank/vacant look, ragdoll appearance and motor incoordination had the highest PPVs while clutching of the head had the lowest PPV. Although the efficacy of the signs in predicting a diagnosis of concussion was similar across studies, the absolute PPVs for each sign appeared to be slightly or moderately higher in the Australian study, likely due to differences in operational definitions, methodology (eg, the primary authors were the coders in the Australian study) and the sports themselves. A careful consideration of these data sets as well as further refinement and consensus regarding the most appropriate means of constructing VS will be useful in the future. For example, it may be that the VS ‘suspected loss of consciousness’ could be parsed into the separate components of failing to brace for impact and lying motionlessness on the ice to improve PPV. Since most VS do not occur in isolation but rather in combination with each other, we examined the PPV of these signs when occurring together. Not surprisingly, sensitivity decreased and specificity increased as the number of signs increased per event. Whereas the PPV of one or more signs per event was 11%, the PPV increased to 100% when five or more signs were present. Similarly, sensitivity, specificity and PPV were maximised when players exhibited one or more of the three VS that showed positive likelihood ratios >1 (LOC, motor incoordination, blank/vacant look). The finding that different combinations of VS have different predictive accuracy should lead to further inquiry as to whether additional factors, such as mechanism of injury, when used in combination with VS could also enhance injury detection.
In conclusion, the use of observable/visual signs to augment early injury detection is useful but not sufficient when applied to players in the NHL. The comprehensive data obtained from this project across 2460 regular season NHL games underscore the complexity of visually identifying concussive injuries. The data reveal that most NHL games do not have players who exhibit VS as currently defined, and some signs are much better predictors of subsequent diagnosis of concussion than others. There do not appear to be any clear pathognomonic VS of concussion. The challenge then is to better identify visual signs, including the possibility of finding additional signs, and to better understand how to use these signs in the context of a multifactorial system of early injury detection. Given the complex nature of this injury it is not surprising that any approach to detection will be equally complex. Importantly, these data only speak to the specifics within the NHL. As we have seen in the Australian data, it is likely that observable signs will differ across sports (eg, American football, rugby, soccer), age of athletes and levels of play. Concentrated efforts should be undertaken to identify commonalities and differences across sports and levels of play in order to enhance injury detection and diminish risk of additional trauma when the brain is most vulnerable.
What are the findings?
Visible signs (VS) of concussion can be reliably identified and coded in professional ice hockey.
The use of observable/visual signs to augment early injury detection is useful but not sufficient since most concussions occur with no VS.
The use of comprehensive approaches to early injury detection is warranted.
How might it impact on clinical practice in the future?
VS of potential concussion can be used to identify players who need to be evaluated for a concussion.
Clinicians can use VS to determine which signs are most predictive of concussion diagnosis.
Organisations may decide to develop spotter programmes to identify signs of concussion.
Acknowledgments
The authors thank the NHL team physicians and athletic trainers/physiotherapists for their continued support and cooperation. These professionals form the essential core of the NHL/NHLPA Concussion Program. The authors also thank the coders in this project who worked tirelessly in a very critical role.
Footnotes
Contributors RJE And JMB took primary responsibility for the design and execution of this study and writing of the manuscript. JMB was involved in the design and execution of this study and in editing the manuscript. WM and MH were involved in the design of the study and contributed to the writing and editing of the manuscript. PC and MA contributed to the design of the study and editing the manuscript.
Competing interests RJE is Co-Chair of the NHL/NHLPA Concussion Subcommittee. He receives financial compensation for consulting services from the NHL, Major League Soccer, US Soccer Federation, NCAA, and Princeton University. JMB is an employee of the NHL. He provides unbranded talks for Novartis and is a member of the Novartis MS and Cognition Medical Advisory Board. He has worked as a consultant to Sporting KC and the Princeton University Department of Athletic Medicine. WM is an employee of the NHL and a member of the NHL/NHLPA Concussion Subcommittee and Co-Chair of the NHL/NHLPA Joint Health and Safety Committee. PC is the co-chair of the NHL/PA Concussion Subcommittee and a consultant to the NHLPA, for which he receives remuneration. MH is a member of the NHL/NHLPA Concussion Subcommittee and a consultant to the NHLPA, for which he receives remuneration.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The data that form the basis for this study are propietary and not available for sharing.