Background Reaction time (RT) is a valuable component of the sport concussion assessment battery. RT is typically measured using computers running specialised software, which limits its applicability in some athletic settings and populations. To address this, we developed a simple clinical test of RT (RTclin) that involves grasping a falling measuring stick.
Purpose To determine the effect of concussion on RTclin and its sensitivity and specificity for concussion.
Materials and methods Concussed athletes (n=28) and non-concussed control team-mates (n=28) completed RTclin assessments at baseline and within 48 h of injury. Repeated measures analysis of variance compared mean baseline and follow-up RTclin values between groups. Sensitivity and specificity were calculated over a range of reliable change confidence levels.
Results RTclin differed significantly between groups (p<0.001): there was significant prolongation from baseline to postinjury in the concussed group (p=0.003), with a trend towards improvement in the control group (p=0.058). Sensitivity and specificity were maximised when a critical change value of 0 ms was applied (ie, any increase in RTclin from baseline was interpreted as abnormal), which corresponded to a sensitivity of 75%, specificity of 68% and a 65% reliable change confidence level.
Conclusions RTclin appears sensitive to the effects of concussion and distinguished concussed and non-concussed athletes with similar sensitivity and specificity to other commonly used concussion assessment tools. Given its simplicity, low cost and minimal time requirement, RTclin should be considered a viable component of the sports medicine provider's multifaceted concussion assessment battery.
- Head injuries
Statistics from Altmetric.com
Sport-related concussion (SRC) is common1 and there is increasing concern regarding possible long-term effects.2–4 While concussed athletes typically recover symptomatically within 1–2 weeks,5 ,6 an athlete whose concussion is unrecognised or who returns to play prematurely is put at risk for more significant injury.7 Thus, prompt recognition of SRC and accurate determination of its resolution are important. However, SRC remains a diagnostic challenge. Presentations are variable and often subtle, and athletes may minimise their symptoms.8 While a growing number of assessment tools have been developed, the diagnosis of SRC remains a clinical one.
Current consensus opinion advocates a multifaceted approach to concussion assessment.9–11 Clinical tools are available to evaluate many aspects of concussion including symptoms,12 neurocognitive function13 and balance.14 Computer-based neurocognitive assessment tools capable of measuring reaction time (RT), attention, working memory and problem solving are also available.15–17 Such tools can be valuable, but their computer-dependence, time requirement and cost limit their applicability in some situations (eg, acute concussion diagnosis) and environments (eg, communities with limited financial resources).
A prolonged RT is common following SRC, and is one of the most sensitive indices of neurocognitive change following injury.18 ,19 In addition RT has been shown to have prognostic value20 and commonly parallels other concussion symptoms.21–24 Moreover, a prolonged RT can persist beyond symptom resolution, suggesting incremental value over solely tracking postconcussive symptoms.5 ,25 ,26
Despite these advantages, computer-independent methods for assessing RT are not available. To address this, we developed a simple, inexpensive method for clinically assessing RT, referred to as clinical RT (RTclin). RTclin is a visuomotor test in which the subject arrests a falling object by hand closure after it is released by an examiner. Pilot work demonstrated the short-term and long-term reliability of RTclin,27 ,28 and its validity with relation to a computerised measure or RT in athletes.29 Furthermore, the task appears to be intrinsically motivating,30 an important characteristic for situations in which an athlete's after-injury performance is compared to preseason baseline.31 Unlike other tests of RT, RTclin has demonstrated functional relevance by showing a strong correlation with an athlete's ability to protect their head in a laboratory simulated athletic environment.32 Recently, we reported a 13.5% prolongation of RTclin in nine collegiate athletes tested within 72 h of SRC as compared with baseline.33 Although the results are suggestive, a larger, controlled study is needed to further elucidate the clinical usefulness of RTclin. Therefore, the purpose of this study is to determine the effect of concussion on RTclin in a larger sample of high school and collegiate athletes through a controlled study design. We hypothesised that RTclin would be prolonged in concussed athletes as compared to baseline and non-concussed control athletes.
Male and female athletes participating in multiple sports at two universities and one high school were recruited during preparticipation physical examinations over two consecutive seasons. Potential subjects were excluded if they were recovering from a concussion or had an upper limb injury preventing the completion of the RTclin task. Adult athletes provided institutional review board (IRB) approved informed written consent and minor athletes provided IRB-approved assent with accompanying parental consent. For the purposes of this study, concussion was diagnosed by the athlete's treating physician as defined in the 2008 Zurich Consensus Statement on Concussion in Sport.9
RTclin was measured in all athletes at baseline during preparticipation examinations. RTclin was determined again within 48 h of injury in athletes who sustained a physician-diagnosed concussion. For each injured athlete, a control athlete was who was a member of the same team and was present in the training room for a reason other than concussion was also retested. Athletes with a dominant upper limb injury preventing completion of the RTclin task were again excluded, analogous to during baseline testing.
The protocol for RTclin testing has been previously described28 ,29 and is illustrated in figure 1. The RTclin apparatus is an 80 cm rigid measuring stick coated in friction tape with a weighted rubber disk affixed to one end. Athletes sat with their dominant forearm resting on a table with their hand positioned over its edge. The hand was open surrounding, without touching, the weighted disk portion of the RTclin apparatus. The examiner suspended the device vertically such that the top face of the disk was aligned in the plane defined by the top of the athlete's open hand. After predetermined randomly assigned delays ranging from 2 to 5 s, the examiner released the apparatus and the athlete caught it as quickly as possible by hand closure. For each trial, an RTclin value (in ms) was calculated from the distance (in cm) that the device fell using the formula for a body falling under the influence of gravity (d=0.5 gt2). After two practice trials, each athlete completed eight data acquisition trials and a mean RTclin value was calculated for the test session. The mean RTclin value was used for analysis. RTclin change scores were determined by the difference between baseline and follow-up values (follow-up RTclin−baseline RTclin) so that a positive RTclin change score represents a decline from baseline, while a negative score represents improvement.
A two-way repeated measures analysis of variance, with paired t tests used post hoc, was used to compare preseason and after-injury RTclin values in concussed versus control athletes. The effect sizes of the RTclin changes observed were described using Cohen's d. Reliable change calculations were performed using SDs of difference scores (SDdiff) from the control group.34 An adjusted reliable change equation, controlling for practice effects,35 was used: where x2−x1 is the individual performance difference between follow-up (x2) and baseline (x1) RTclin in a concussed athlete; μ2−μ1 is the mean group difference between follow-up (μ2) and baseline (μ1) RTclin in the control group; and SDdiff is the SD of the change scores for the control group. An adjusted reliable change index has been demonstrated to perform comparably to more complex regression-based formulas36–38 and has been advocated over regression-based methods.36
Reliable change values can be interpreted by comparing them to the z-distribution critical value associated with a desired one-tailed probability of error. One-tailed critical values were used because the outcome of interest was a decline in performance from baseline, rather than any difference, which could represent either decline or improvement.39 ,40 For example, a critical value of 1.65 is used when a 5% type I error rate is deemed acceptable. Therefore a reliable change score greater than 1.65 can be interpreted as an indication that an athlete has experienced a significant decline in their RTclin performance, with a 5% probability of type I error.
Conversely, once the mean RTclin change score and its associated SD are calculated in the control group, a cut-off value can be generated indicating the RTclin change score that represents significant change at a given level of confidence. We calculated reliable change cut-off values over a range of confidence levels and determine the combined sensitivity and specificity associated with each value.41 In addition, a receiver operating characteristic curve (ROC curve) was generated to compare the sensitivity and specificity associated with cut-off value over the range of RTclin change scores observed. Sensitivity and specificity values were summed at each cut-off value, with the highest summed score interpreted as having the greatest combined sensitivity and specificity.41 ,42
Twenty-six athletes sustained 28 concussions (with two athletes sustaining repeat concussions) over the 2-year study. The majority of concussions occurred in male collegiate football players (n=20, 71.4%). The remaining injuries were sustained by athletes participating in male high school football (n=2, 7.1%), female collegiate soccer (n=2, 7.1%), male collegiate ice hockey (n=1, 3.6%) and male high school ice hockey (n=1, 3.6%). Of note, two injuries which occurred during year 2 involved athletes who did not have baseline RTclin results available from that year. Therefore, year 1 baseline RTclin results were used for these subjects and for their matched controls. Eight of the postconcussion testing sessions were completed on the day of injury, 12 were completed 1 day after injury and eight were completed 2 days after injury not exceeding 48 h. All concussed athletes were still reporting postconcussion symptoms at the time of testing. The time from baseline to follow-up RTclin testing did not differ between the concussed and control groups (mean±SD=60.5±40.5 and 73.8±51.0 days, respectively; p=0.631).
Concussed and control athletes demonstrated nearly identical mean (±SD) baseline RTclin values (203±22 ms vs 202±16 ms, respectively; p=0.839). After-injury RTclin values were significantly greater (longer) as compared with baseline (220±29 ms; p=0.003; effect size, d=0.616), while there was a trend towards shorter RTclin values in the 28 controls (193±17 ms, p=0.058; effect size, d=−0.373). The group by test interaction was highly significant (p<0.001). The contrast between baseline and follow-up RTclin in the two groups is illustrated in figure 2.
Reliable change cut-off values for RTclin and the associated sensitivity and specificity at each confidence level are presented in table 1. The cut-off values represent the RTclin change values necessary to conclude that an athlete has experienced a significant prolongation in RTclin at a given level of confidence. An ROC curve comparing the sensitivity and specificity of RTclin at every possible cut-off value over the observed range of RTclin change values was also generated (figure 3). Resulting analysis demonstrated that cut-off values from 0 to −2 ms (ie, 2 ms of improvement at follow-up) maximised sensitivity and specificity of RTclin with a combined value of 1.43. This corresponds to a critical value of 0.375 and an associated one-sided confidence level of 65%.
In this study we observed a significant decline in RTclin performance following SRC in a group of 28 concussed athletes (8.4% slower RT). These results are similar to those previously reported.33 The effect size reported here is generally considered to be moderate in size.43 In contrast, follow-up RTclin performance showed a trend towards improvement in the control athletes. Group analyses showed that the contrast between postinjury performance declines observed in the injured athletes versus the improvement in controls was highly significant.
Reliable change analyses allow group data to be applied to individuals. We found that to be 95% confident an athlete's RTclin performance has declined, RTclin must be at least 31 ms slower during follow-up testing. Using this cut-off value correctly identifies 11 of the 28 concussed athletes studied (39% sensitivity), but it incorrectly identifies just 2 of the 28 control athletes as having performed worse (93% specificity). Conversely, if we choose to be 60% confident that an athlete's follow-up RTclin performance is worse the cut-off value is 3 ms improvement from baseline, due to the non-concussed athletes having faster RTclin at follow-up. These control athlete data suggest a small learning effect, consistent with prior observations.27 Using this 3 ms improvement cut-off value correctly identifies 22 of the 28 concussed athletes (79% sensitivity), while incorrectly identifying 11 of the 28 control athletes (61% specificity).
Screening tests should have high sensitivity so as to identify the majority of affected individuals. Given the inherent trade-off between sensitivity and specificity, an increased sensitivity invariably results in an increased false-positive rate. However, given the current ‘when in doubt sit them out’ approach with regard to concussion, this trade off seems appropriate. This is especially true in younger athletes, where the consequences of a missed injury are greatest.7 Moreover, maximising test sensitivity at the expense of specificity has been advocated by other groups with regard to concussion assessment tools.42 ,44–47 In our study, a change score of 0 ms (ie, interpreting any follow-up RTclin value, that is, slower than baseline as abnormal) was found to maximise the combined sensitivity (75%) and specificity (68%) of the test and correlated with a reliable change confidence level of 65%. Therefore we propose an RTclin cut-off value of 0 ms be used when screening an athlete with suspected concussion.
The test characteristics for RTclin reported here are similar to other recognised concussion assessment tools. For example, McCrea et al47 reported on the sensitivity and specificity of four commonly used assessment tools. Test sensitivities were highest when testing occurred within 3 h of injury, with the Graded Symptom Checklist (GSC) being the most sensitive (89% sensitive, 100% specific), followed by the Standardized Assessment of Concussion (SAC; 80% sensitive, 91% specific), and the Balance Error Scoring System (BESS; 34% sensitive, 91% specific). The neuropsychological test battery administered 2 days following injury had a sensitivity and specificity of 23% and 93%. At the same 2 day postinjury interval the sensitivity/specificity of the GSC, SAC and BESS dropped substantially to 27%/100%, 22%/89% and 24%/91%, respectively. Overall a marked reduction in sensitivity was observed for each test over the first week following injury. The authors also noted that combinations of tests had greater sensitivity than individual tests. Broglio et al48 later reported on the sensitivity of five different concussion assessment tools.48 The most sensitive measure was the Immediate Post-concussion Assessment Cognitive Test (ImPACT; 79.2%), with its most sensitive component being the symptom inventory (62.5%). The combined sensitivity of the remainder of the ImPACT battery was 62.5%, with the memory and RT composite scores being the most sensitive individual tests (41.7% each). The overall sensitivity of the HeadMinder Concussion Resolution Index was 78.6%, with individual test component sensitivities of 71.4% for simple and complex RT and 50% for processing speed. A nine item concussion symptom scale was the next most sensitive tool (68%). As was the case in McCrea's study, test batteries demonstrated greater sensitivity than any single component used in isolation. The specificities of these tests were not reported. Other relevant studies in this regard are summarised in table 2.
In aggregate, the sensitivity and specificity of concussion assessment tools are greater when the test is conducted immediately following injury, with a gradual decline in sensitivity over time. It also appears that test batteries with multiple components are more sensitive than individual measures. With these considerations in mind, it is remarkable that RTclin, a single 5 min test that was performed up to 48 h after injury with equipment costing less than five US dollars, has comparable test characteristics to other concussion assessment tools currently in use, including lengthier computerised neuropsychological test batteries composed of multiple individual tests (eg, ImPACT and CRI), traditional paper and pencil neuropsychological test batteries (composed of the HVLT, TMT, SDMT, DS, COWAT and Stroop), and costly balance assessment tools (eg, SOT). Further research should evaluate the test characteristics of a concussion assessment battery composed entirely of clinical assessment tools such as a symptom checklist, the SAC, RTclin and the BESS. Such a battery is likely to have improved sensitivity over its individual component measures, and could easily be used by sports medicine providers in nearly any location at minimal expense.
The control athletes represent one of the study's strengths. These athletes were retested at similar time intervals as their concussed team-mates and were typically receiving treatment for non-concussive injuries at the time of follow-up assessment. This is important and suggests that the concussed athletes’ altered RTclin performance is more likely related to their concussions than to other in-season factors affecting the team that were not measured or directly controlled for in our study design. In addition, the results are more compelling given prior work finding RTclin to be valid, intrinsically motivating and functionally relevant.27–30 ,32 The study also has limitations that merit discussion. The majority of concussed athletes were male collegiate football players, and so the generalisability of this study's findings to other populations is uncertain. Also, the athletes underwent after-injury testing at a single time point within 48 h of concussion, and so no information regarding a postinjury ‘recovery curve’ is available. Furthermore, since additional convergent and divergent measures were not simultaneously assessed during this study and final athlete outcomes were not recorded, this study does not fully address the convergent, divergent or predictive validity of RTclin for concussion. In addition, in some cases the follow-up tests on control athletes were not conducted on the exact same day as the after-injury tests in their concussed team-mates. We do not suspect that these generally small differences in time until follow-up testing have introduced significant bias, however their effect is not known. Finally, examiners were not blinded to athletes’ concussion status during follow-up testing and so the introduction of measurement bias is possible. This is less likely given the nature of the task, which occurs on a millisecond scale, and given that examiners were not aware of the athlete's baseline performance.
In conclusion, the results suggest that RTclin performance is impaired following concussion, while in-season follow-up RTclin testing of non-concussed (but often injured) control athletes is associated with a small learning effect. Reliable change calculations indicate that a critical RTclin change score of 0 ms (ie, interpreting any decline from baseline as indicative of impairment) is 75% sensitive and 68% specific for concussion. These test characteristics compare favourably with most currently available concussion assessment tools, many of which are associated with significantly more time, equipment, and expense than RTclin. Furthermore, RTclin may have even greater utility when used as part of a concussion assessment battery, in concert with a symptom assessment, and clinical cognitive and balance assessments. RTclin, like any concussion assessment tool, cannot diagnose concussion in isolation and must be interpreted by a knowledgeable sports medicine provider in the greater clinical context of the athlete being assessed. The results of this study support the potential use of RTclin as part of a multifaceted clinical concussion assessment battery.
In this study, clinically measured reaction time (RT) distinguished between concussed and non-concussed athletes.
When any increase in clinically measured RT compared with the athlete's own baseline was considered abnormal (ie, a critical cut-off value of 0 ms was used), the test sensitivity and specificity were 75% and 68%, respectively.
These findings support the use of clinically measured RT as part a multifaceted sports medicine concussion assessment battery.
The authors would like to thank all of the athletic trainers, team physicians, residents and students at Eastern Michigan University, the University of Michigan, and Ann Arbor Skyline and Pioneer High Schools who assisted with data collection for this study. Special thanks go to Steve Nordwall, Paul Schmidt, Rick Bancroft, Lenny Navitskis, Phil Johnson, Melissa Pohorence, Bill Shinavier, Jennifer Garcia, Dave Cotner, Allyson Calhoun, Nate Santoni, Shailesh Reddy, Andy Schuldt and Tyler Ladue. We would also like to thank all of the athletes and coaches whose teams participated in this study. Dr Eckner also thanks the Rehabilitation Medicine Scientist Training Program for its support of his effort on this project through a K-12 career development award.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Contributors Dr JTE contributed to the study conception and design, as well as the analysis and interpretation of data. He drafted the article and provided final approval of the submitted manuscript. Dr JSK contributed to the study conception and design, as well as the interpretation of data. He revised the article critically for important intellectual content and provided final approval of the submitted manuscript. Dr SPB contributed to the analysis and interpretation of data. He revised the article critically for important intellectual content and provided final approval of the submitted manuscript. Dr JKR contributed to the study conception and design, as well as the analysis and interpretation of data. He revised the article critically for important intellectual content and provided final approval of the submitted manuscript.
Funding Dr Eckner received a K-12 career development award from the Rehabilitation Medicine Scientist Training Program which supported his effort on this project (grant number 5K12HD001097).
Competing interests An ICMJE conflict of interest form has been submitted by each author.
Ethics approval University of Michigan-IRBMED.
Provenance and peer review Not commissioned; externally peer reviewed.
▸ References to this paper are available online at http://bjsm.bmj.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.