Background Sideline detection is the first and most significant step in recognising a potential concussion and removing an athlete from harm. This systematic review aims to evaluate the critical elements aiding sideline recognition of potential concussions including screening tools, technologies and integrated assessment protocols.
Data sources Bibliographic databases, grey literature repositories and relevant websites were searched from 1 January 2000 to 30 September 2016. A total of 3562 articles were identified.
Study selection Original research studies evaluating a sideline tool, technology or protocol for sports-related concussion were eligible, of which 27 studies were included.
Data extraction A standardised form was used to record information. The QUADAS-2 and Newcastle-Ottawa tools were used to rate risk of bias. Strength of evidence was assessed using the Grades of Recommendation, Assessment, Development and Evaluation Working Group system.
Data synthesis Studies assessing symptoms, the King-Devick test and multimodal assessments reported high sensitivity and specificity. Evaluations of balance and cognitive tests described lower sensitivity but higher specificity. However, these studies were at high risk of bias and the overall strength of evidence examining sideline screening tools was very low. A strong body of evidence demonstrated that head impact sensors did not provide useful sideline concussion information. Low-strength evidence suggested a multimodal, multitime-based concussion evaluation process incorporating video review was important in the recognition of significant head impact events and delayed onset concussion.
Conclusion In the absence of definitive evidence confirming the diagnostic accuracy of sideline screening tests, consensus-derived multimodal assessment tools, such as the Sports Concussion Assessment Tool, are recommended. Sideline video review may improve recognition and removal from play of athletes who have sustained significant head impact events. Current evidence does not support the use of impact sensor systems for real-time concussion identification.
- sports related concussion
- diagnostic accuracy
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Despite a consensus definition of sports-related concussion (SRC) having been well elucidated,1 its immediate and accurate recognition in a clinical setting remains a challenge.2 Sustaining a SRC may increase the likelihood of incurring a subsequent head or musculoskeletal injury,3 and repeated concussions could be associated with long-term consequences such as persistent postconcussive symptoms, depression or neurodegenerative disorders.1 4 5 Early detection of suspected concussion and removal of the affected player will help prevent these potential adverse sequelae and facilitate further evaluation, management and safe return to play. This systematic review aims to evaluate the critical elements aiding off-field (commonly termed ‘sideline’) recognition of potential concussions. Specific objectives were to assess the diagnostic accuracy of existing clinical screening and diagnostic tools, determine the utility of technology in detecting SRC and assess integrated head injury assessment protocols currently used in professional collision sports.
Expert consensus guidelines for the conduct of systematic reviews were followed,6–8 and a detailed protocol stating an a priori analysis plan was registered before data collection (PROSPERO 2016:CRD42016037831). The review question and inclusion/exclusion criteria are detailed in table 1. Online supplementary details on methodology, including a glossary of technical terms, are presented in the online supplementary file web appendix.
Identification of evidence
An extensive range of electronic information sources were examined including all major bibliographic databases, specialist sports medicine databases, grey literature repositories and relevant websites (see online supplementary web appendix for details). Additional information sources included forward and backward citation searching, author searching, reference checking and contact with experts. Search strategies for bibliographic databases were developed iteratively in conjunction with an information services specialist (University College London) and underwent external peer review (University of Sheffield).
Searches were conducted for original research published between 2000 (corresponding to the modern definition of concussion) and week 4, April 2016, and were otherwise unrestricted. Current awareness searches were conducted in MEDLINE and Embase (week 4, September 2016) immediately prior to submission.
Selection of evidence and data extraction
Original research studies identified during searches were assessed in a four-stage process by teams of two independent reviewers. First, titles and abstracts were screened for relevance. Second, full-text articles were examined as required to assess eligibility. Third, studies meeting review inclusion criteria were classified into domains pertaining to: sideline screening tests (comprising subtopics of clinical signs and symptoms, balance tests, oculomotor assessments, cognitive tests and multimodal testing strategies); technology; and professional sports-specific head injury assessment protocols (defined in table 1). Finally, data extraction was performed separately for eligible studies within each subtopic. A single unblinded reviewer extracted information on study characteristics, methodology and results using a standardised data extraction form; and a second reviewer independently checked data for consistency and accuracy. In cases of disagreement at any stage, consultation with a third author was planned, with consensus derived by arbitration.
Risk of bias assessment
Included studies were assessed for risk of bias using peer-reviewed critical appraisal checklists appropriate to study design. The QUDAS-2 tool was used for diagnostic accuracy studies.9 Observational studies were evaluated using the Newcastle-Ottawa scale.6 A single unblinded reviewer within each subgroup team assessed risk of bias, with a second reviewer independently checking the assessment for validity. Any disagreement between reviewers was resolved by consensus and consultation with a third author with expertise in epidemiology and critical appraisal.
Data synthesis, statistical analyses and assessment of overall quality of evidence
Data synthesis and statistical analysis were performed separately for eligible studies within each subtopic. Results are presented descriptively with reported point estimates and 95% CIs and summarised graphically using Forest plots.10 Heterogeneity was assessed using the I2 statistic.11 A narrative synthesis was prespecified in the event that clinically and methodologically homogenous studies at low risk of bias were not identified. References were managed in EndNote (Clarivate Analytics, Berkeley California, USA), extracted data were collated in Excel 2013, and Forest plots were formulated using Meta-DiSc V.1.4 (University of Birmingham, Birmingham, UK). The overall quality of evidence for each outcome was assessed using the consensus Grades of Recommendation, Assessment, Development and Evaluation Working Group (GRADE) approach.12 GRADE is a systematic method of assessing quality of evidence and strength of recommendations taking into account methodological flaws, consistency of results, generalisability of findings and the effectiveness of treatments. A clinical diagnosis of concussion was the primary outcome for each domain.
A total of 3562 citations were screened for eligibility, with the full text of 198 articles retrieved for detailed evaluation. During full text examination, 27 studies were found meeting review inclusion criteria: sideline screening assessment (21 studies); integrated diagnostic protocols (1 study) and technology (5 studies). Figure 1 describes the selection of studies in detail.
Sideline screening tests
Characteristics of included studies
Twenty-one studies met review inclusion criteria and reported interpretable data on the diagnostic accuracy of screening tests, either alone or in combination, to identify suspected SRC. Characteristics of the included studies examining sideline assessments are summarised in table 2.
Risk of bias
Assessment of risk of bias is summarised according to QUDAS-2 domains in table 3 and figure 2. Overall risk of bias was high or unclear for all included studies. The predominant limitation was the use of a ‘two-gate’ study design using healthy controls, which is known to overestimate estimates of test performance.13 14 Other systematic errors included delayed index testing, inaccurate reference standard assessment by a non-medically trained outcome assessors, and test and diagnostic review, incorporation and attrition biases. Detailed risks of bias evaluations are presented in the online y supplementary web appendix.
The diagnostic accuracy of sideline assessments for detecting suspected concussion is summarised in figure 3. Studies examining symptoms, the King-Devick (KD) test and multimodal assessments reported relatively good sensitivity and specificity. Evaluations of balance and cognitive tests described lower sensitivity, but relatively good specificity. However, results were imprecise and heterogeneous for all types of sideline assessments, in addition to the concerns regarding the internal validity. The overall quality of evidence according to GRADE criteria was very low for all classes of sideline tests based on serious concerns regarding inconsistency, imprecision and risk of bias. Detailed results and evaluation of overall quality of evidence for individual tests are provided in the online supplementary web appendix.
Five studies met review inclusion criteria and reported interpretable data on the use of a technology in sideline screening for SRC, examining head impact sensors (four studies) and sideline video review (one study).15–19 Overall risk of bias was low for all studies. Reported results indicated that no clinically significant relationship existed between impact magnitude, or location, and concussion (p>0.05). Fuller et al16 reported that sideline video review contributed to identification of 61.5% of significant head impact events and influenced sideline evaluation in 20.4% of cases. The overall GRADE quality of evidence was rated as high for head impact sensors and low for sideline video review. Table 4 summarises the characteristics, risk of bias and main results of included technology studies. Further details on risk of bias and GRADE ratings are provided in the online supplementary web appendix.
Integrated head injury assessment protocols
No experimental or comparative effectiveness research was identified evaluating the performance of alternative head injury assessment protocols. However, a single study at low risk of bias was retrieved which evaluated a comprehensive system used at the elite level in Rugby Union.16 The major finding was the importance of a multimodal, multitime-based concussion evaluation process incorporating video review to identify significant head impact events and delayed onset concussion. The overall GRADE quality of evidence was rated low, secondary to imprecision and potential inconsistency. Further details on existing integrated head injury assessment protocols in professional sports, and the characteristics of Fuller et al are provided in the online supplementary web appendix.
Summary of key findings
Studies examining symptoms, the KD test and multimodal assessments reported high sensitivity and specificity. Evaluations of balance and cognitive tests described lower sensitivity, but good specificity. However, the overall strength of evidence examining sideline screening tools was of very low quality secondary to high risk of bias, and imprecise and heterogeneous diagnostic accuracy estimates. Studies examining technology provided a high (head impact sensors) or low (video review) strength body of evidence. Head impact sensors did not provide useful information. Conversely, a multimodal, multitime-based concussion evaluation process incorporating video review appeared to be important for the identification of significant head impact events and the delayed onset concussion.
A meta-analysis was not performed due to the absence of studies at low risk of bias and marked heterogeneity; in accordance with the prespecified analysis plan, a narrative synthesis was therefore conducted. Interestingly, no obvious patterns were evident between study results and design characteristics including sample size, setting, performance level, sport or risk of bias. This may be due to the inherent generalisability of findings, but could also be explained by biases operating in different directions and to varying magnitudes across different studies.
Notwithstanding the high risk of systematic error, a wide range of settings, sports and age groups were investigated in eligible studies suggesting good external validity of findings. However, in addition to information on diagnostic accuracy, the feasibility, cost and acceptability of alternative sideline tests may be important in applying these results to different settings. The availability of baseline data, testing environment and influence of the athlete-physician relationship could also affect generalisability. Importantly, in lower levels of competition where medical staff may be limited, an alternative ‘recognise and remove’ approach is recommended, with exclusion of the sideline screening stage, and immediate and permanent removal from any further participation when there is any suspicion of concussion.1 20
A key concept in sideline assessment is the rapid screening for a suspected concussion, rather than the definitive diagnosis of a head injury. Players manifesting clear on-field observable signs, such as loss of consciousness, ataxia, tonic posturing or post-traumatic seizures, can immediately be diagnosed with a concussion and removed from sporting participation. Athletes with the possibility of suspected concussion following a significant head impact event can alternatively proceed to sideline screening, with a later definitive diagnostic evaluation. Clearly, to allow sufficient time and a suitable environment for testing, this should occur away from the sporting environment, and may necessitate a temporary athlete interchange. The importance of off-field testing is exemplified by findings in professional Rugby where the number of players with confirmed concussion returning to play following their head injury dropped from 56% to 13% following the introduction of the Pitchside Suspected Concussion Assessment that superseded an ‘on-the-field-and-on-the run’ approach.21
Elite contact and collision sports are played at a fast pace in a disorganised environment, where the view of medical staff may be obscured, challenging the evaluation of head impact events. Video review appeared to be helpful in identifying both observable signs of concussion and cases of possible suspected concussion where further assessment off-field is beneficial. Furthermore, evolving and delayed onset concussions have been well described,16 22 highlighting the importance of careful follow-up after a significant head impact, regardless of a negative sideline screening test or early diagnostic evaluation. Consequently, implementation of systematic head injury assessment protocols appears to improve detection and management of the full spectrum of SRC.
Concussion can manifest as a diverse range of somatic, cognitive, behavioural or emotional symptoms; and/or physical signs such as vestibulo-ocular deficits, loss of consciousness and ataxia.1 It would therefore be expected ex ante that multimodal assessments, evaluating several of these domains, are necessary to maximise detection of different subtypes of SRC. However, with simultaneous testing a net gain in sensitivity usually occurs at the expense of a net loss in specificity.23 Interestingly, included multimodal assessment studies reported both high sensitivity and specificity which could suggest either an optimal combination of tests, or could reflect study biases. Given the absence of definitive evidence on the performance of sideline tests, expert consensus opinion is necessary to guide practice and strongly recommends the use of a multimodal assessment tool, of which the Sports Concussion Assessment Tool (SCAT; now in its 4th version) is the most established, well developed and studied.24
It is important to note that the pretest probability of concussion will strongly influence the performance of sideline screening tests.25 In settings with high prevalence of concussion, or high test thresholds, the negative predictive value of sideline tests will fall. High sensitivity and specificity would consequently be necessary to ensure the detection of a satisfactory proportion of cases. Conversely, indiscriminate testing, with a lower pretest probability of concussion, would result in higher negative predictive values, but worsening numbers of false positives. Such a safety first approach might be preferred in non-elite settings.
Consistency with other studies or reviews
There have been a large number of narrative reviews, position statements and editorials that have previously examined the role of sideline screening tests or technology in the detection of SRC. Although these articles are inherently limited by a lack of defined inclusion criteria, systematic search strategies and transparent risk of bias assessment, their conclusions are broadly consistent with the current systematic review. For example, Eckner et al 26 stated that ‘multiple assessment tools are available, with no single tool showing clear superiority. Many tools remain based more on expert opinion than rigorous scientific evaluation.’
Six related systematic reviews were also identified during the literature searches, comprising examination of symptom checklists,27 SCAT 2/3,28 Balance Error Scoring System,29 KD test30 31 and sideline testing in general.32 Although the review questions were not directly comparable, including delayed non-sideline testing and additional examination of test reliability, similar studies were often included and conclusions concurred with the current study in Alla,27 Yengo-Khan,28 Bell29 and Hunt.30For example, Alla27 noted that ‘There is little information available on the derivation or psychometric properties (eg, sensitivity, reliability, etc) of the various symptom scales’, and Yengo-Khan28 observed that ‘the sensitivity and specificity of the SAC has been reported sparsely.’ Conversely, in contrast to the current findings, Galetta31and King32 concluded that the KD test can successfully identify SRC on the sideline. This divergent opinion is explained by the absence of any risk of bias assessment for constituent studies included in their reviews, resulting in the KD test being interpreted as having high sensitivity and specificity whereas the quality of evidence presented did not justify this.
Implications for research
There is an absence of valid research confirming the diagnostic accuracy and impact on improving outcomes of currently used sideline screening tests. Adequately powered diagnostic accuracy studies are therefore recommended that enrol a representative sample of athletes with suspected concussion following non-trivial head impact events. Ideally, once the diagnostic accuracy and optimal threshold of sideline tests have been confirmed, comparative effectiveness studies would investigate whether important outcomes are improved. Further research is also recommended to investigate the impact of integrated head injury assessment protocols and sideline video review for the evaluation of head impact events. Further research could usefully examine novel sideline screening tests such as reaction times, or investigate the utility of tablet software applications as an adjunct to sideline concussion screening.
There are a number of potential methodological weaknesses which could limit the validity of this systematic review. Because of time constraints, hand searching of journals and conference proceedings was not performed and regional bibliographic databases were not included raising the potential for publication bias. Decisions on study relevance, information gathering and validity were unblinded and potentially could have been influenced by preformed opinions. Furthermore, data extraction and risk of bias assessment were not performed in duplicate (ie, two truly independent reviews), with the second reviewer checking the assessment of the first reviewer. Finally, assessment of reference standard bias was challenged by the lack of a convincing diagnostic gold standard.
Based on this systematic review of the literature, an evidence-based recommendation for any individual screening test or protocol is not possible. The recognition of suspected concussion is therefore best approached using multimodal testing guided via expert consensus. The SCAT currently represents the most well-established and rigorously developed instrument available for sideline assessment. The addition of video review could potentially offer a promising approach to improve identification and evaluation of significant head impact events, and a multitime-based concussion evaluation process appears to be important to detect delayed onset SRC. The KD test shows promise as a sideline screening test but requires adequately powered diagnostic accuracy studies which avoid a two-gate design with healthy controls, and enrols a representative sample of athletes with suspected concussion. Collaboration between sporting codes to rationalise multimodal diagnostic sideline protocols may help facilitate more efficient application and monitoring. Current evidence does not support the use of impact sensor systems for real-time concussion screening.
Competing interests JP received travel subsidies for conferences from South African Rugby and World Rugby. GWF is funded by the National Institute for Health Research and received travel funding from World Rugby. RE, SH, JSK, ML, MM, MP have no competing interests to decalre. MM received travel and accommodation costs. KJS has received speaking honoraria for presentations at scientific meetings.
Provenance and peer review Not commissioned; externally peer reviewed.