Background ‘On the field and on the run’ assessments of head impact events in professional rugby have resulted in a high proportion of players subsequently diagnosed with confirmed concussion not leaving the field of play at the time of injury. The International Rugby Board (IRB) consequently developed a process to support team doctors in the recognition and management of players at risk of concussion, including development of a multimodal assessment instrument—the Pitch Side Concussion Assessment (PSCA) tool.
Methods This was a pilot cohort study designed to determine the feasibility of assessing the accuracy of the IRB PSCA tool in elite male rugby. The study population comprised consecutive players identified with a head impact event with the potential to result in concussion during eight international/national competitions. Players were assessed off field by match-day or team doctors, following a temporary substitution. The accuracy of the PSCA tool was measured against a reference standard of postmatch confirmed concussion, based on clinical judgement aided by an established concussion support instrument.
Results A total of 165 head injury events with the potential to cause concussion were included in the study. The PSCA tool demonstrated a sensitivity of 84.6% (95% CI 73.5% to 92.4%) and a specificity of 74% (95% CI 64.3% to 82.3%) to identify players subsequently diagnosed with confirmed concussion. The negative predictive value was 88.1% (95% CI 79.2% to 94.1%); the positive predictive value was 67.9% (95% CI 56.6% to 77.8%). There were no major barriers identified that would prevent the evaluation of the PSCA process or tool in a future large-scale study.
Conclusions This pilot study has provided the first preliminary estimates for the performance of the PSCA tool, suggesting a potentially favourable balance between positive and negative predictive values. The study has also offered a strong basis to conduct a further larger trial, providing information for sample size calculations and highlighting areas for methodological development.
- Head Injuries
- Injury Prevention
Statistics from Altmetric.com
Concussion represents a subset of traumatic brain injury, pragmatically defined as “a traumatically induced transient disturbance of brain function caused by a complex pathophysiological process.”1
The management of head impact events with the potential for concussion is a prominent and contentious issue in collision and full-contact sports.2 ,3 Players who immediately return to play following a concussive event may risk further injuries and exacerbation of concussive symptoms.2
Although controversial, it has also been suggested that a catastrophic diffuse cerebral swelling can occur rarely following a successive ‘second impact’.4 In addition to these early sequelae, possible causal links between repetitive concussive episodes and increased risk of depression,5 chronic traumatic encephalopathy6 and other neurodegenerative diseases have been proposed.7
Concerns about sport-related concussion were addressed in the recent consensus statement from the 4th International Conference on Concussion in Sport,8 and in a position statement from the American Medical Society for Sports Medicine,1 aiming to mitigate the risk from concussive events. The importance of a multimodal assessment, with subsequent removal from play for any athlete with suspected concussion, is emphasised in both publications. However, it was acknowledged that the accuracy and reliability of the current on-pitch and sideline assessment tests are ‘largely undefined’.
Rugby is a highly physical, dynamic, full-contact sport with an estimated incidence of concussion between 4 and 9 injuries/1000 player-match hours in professional competitions.9–12 Despite this burden, the acute management of concussion in elite level rugby may be suboptimal, typically involving brief ‘on the field and on the run’ assessments. For example, the English Rugby Football Union reported that between 2002 and 2006, 58% of confirmed concussion cases were not removed from the field of play despite medical practitioner education.9 The challenges of assessing players with a potential head injury in rugby are well recognised. The laws of the game do not allow free player interchanges for off field appraisal of injuries and the in-game environment may limit the evaluation of important clinical domains or distract players during assessments. In addition, there is concern about the potential for rule manipulation, which was previously observed following the introduction of temporary substitutions for ‘blood injuries’.13
Consequently, a Pitch Side Concussion Assessment (PSCA) working group was convened by the International Rugby Board (IRB) to improve the management of head injuries and concussion in elite rugby. At the initial meeting of this working group, it was recognised that concussion was a complex injury, difficult to diagnose and assess, and that no validated diagnostic test existed for an immediate conclusive diagnosis. Informed by recent consensus statements, a PSCA process was developed to support team doctors in the recognition of a player at risk of concussion, including a multimodal assessment instrument for the pitchside (rather than on-pitch) assessment of concussion—the PSCA tool.
The aim of the pilot study reported here was to determine the feasibility of examining the validity of the PSCA tool in a definitive investigation. Specific objectives were to identify barriers for implementing a large-scale study; estimate the prevalence of concussion within the target population to guide future sample size calculations and obtain a preliminary estimate of the sensitivity and specificity of the PSCA tool. Implementation of the PSCA tool was also evaluated in secondary analyses.
Study design and setting
This was a pilot cohort study designed to determine the feasibility of assessing the accuracy of the IRB PSCA tool in elite male rugby. To maximise internal validity, the study followed expert recommendations on the conduct and reporting of pilot and diagnostic accuracy studies.14 ,15
The source population consisted of elite rugby players participating in eight international/national competitions between October 2012 and July 2013, comprising: The Rugby Championship, Northern hemisphere autumn test matches, ITM Cup, Aviva Premiership, Currie Cup, Super Rugby, Top 14 and ProD2. Further details on participating competitions are provided in web appendix 1. The subsequent study sample included consecutive players identified during play by match officials, team doctors or match-day doctors as having sustained a head impact event, either a direct blow or indirect transmission of force, with the potential for a concussion. Players with immediate, clear clinical signs or symptoms of concussion (eg, confirmed loss of consciousness, tonic posturing, convulsions) were excluded.
The index test under consideration was the IRB PSCA tool (see web appendix 1), an assessment instrument designed to support clinicians in identifying suspected concussion and guide their clinical decisions on removal from play following a head impact event as part of the IRB PSCA process. This instrument includes assessment of four sequential domains, comprising the Maddocks’ questions,16 a tandem balance test,17 a symptom assessment and a brief evaluation of mental status. An affirmative finding in any one of these aspects is regarded as a positive test result indicating suspected concussion, which mandates removal of the player from the game. Negative test results were interpreted within the context of overall clinical judgement, and did not automatically mean a return to play.
The PSCA tool had a central role within the wider PSCA process, summarised in figure 1. Testing with the PSCA tool could be requested by the match referee, team doctor or independent match-day doctor following a head impact event (either a direct blow or indirect contact with transmission of impulsive force to the head) with the potential for a player to sustain a concussion. Players with immediate, clear clinical signs or symptoms of concussion were directly removed from play and thus not subjected to assessment with the PSCA tool. Additionally, the PSCA process also allowed players to be permanently removed from the game if a player was clinically suspected of having a concussion, regardless of a negative PSCA test result.
The team doctor had the primary responsibility for conducting tests with the PSCA tool but could delegate this role to the match-day doctor. The test was administered in the stadium's medical room or other agreed venue if this site was too distant from the field. The PSCA process had to be completed within 5 min (absolute time, not playing time), starting when a player left the field of play and finishing when the player presented to the fourth match official for permission to return to play. A temporary player interchange was allowed during the assessment period.
The overall framework for assessing players with the potential for concussion, and the components of the PSCA tool, were formulated following a review of the scientific literature, consideration of position statements and consultation with sports medicine doctors, coaches, players and referees. The 5 min assessment period was considered sufficient time to safely complete all steps in the PSCA process and judge whether a player had a suspected concussion, with minimal disruption to play in the event of a negative test.
The reference standard, against which the accuracy of the PSCA tool was tested, was the ultimate clinical diagnosis of concussion determined postmatch with the help of an established concussion support instrument. All players evaluated by the PSCA tool underwent confirmatory postgame testing with the reference standard by their team doctors. The choice of the postgame concussion support tool was at the discretion of individual team doctors and the timing of the application varied from immediately postmatch to 48 h postinjury. Reference standard assessors were not blinded to the results of the PSCA tool.
Index test findings were recorded contemporaneously using standardised paper forms. Concussion confirmatory test results were also immediately recorded by the administrating clinician, with exact documentation varying according to the specific support instrument used. All data were subsequently collected using a central IRB web-based data portal by specific competition coordinators. All data were anonymised.
Primary analysis: accuracy of the PSCA tool
The accuracy of the PSCA tool was assessed in the primary analysis. Sample characteristics were initially examined using descriptive statistics. Each case was then coded according to the index test and reference standard result, with a 2×2 contingency table constructed to determine true positives, false positives, true negatives and false negatives. Prevalence of concussion, sensitivity and specificity, with their 95% CIs, were subsequently calculated. The characteristics of false-negative cases were compared descriptively with true-positive cases to explore the determinants of test sensitivity. The influence of each subcomponent of the PSCA tool on overall accuracy was also examined. Sensitivity and specificity for identifying concussion were computed separately for each of the four individual domains of the assessment instrument.
Secondary analyses: implementation of the PSCA tool
Implementation of the index test was investigated by examining the concordance of observed decisions on removal from play with those recommended by the PSCA tool. Sensitivity and specificity were subsequently recalculated to estimate accuracy for reported removal from play decisions, rather than those suggested by PSCA tool results alone, and for the hypothetical contingency of allowing clinician discretion to overrule negative PSCA tool results only.
Statistical analysis, ethics and funding
In the absence of unique patient identifiers, it was assumed that each observation was independent with no player sustaining more than one head impact event. A conventional significance level (α) of 0.05 was used in all analyses. Statistical analyses were carried out in Stata V.12.1 (StataCorp, College Station, USA). Agreement with study processes, affirmation of informed consent from study participants and permission for the use of anonymised data were formally confirmed with the presiding medical officers of each participating competition. Research funding was provided by the IRB.
Derivation and characteristics of study participants
A total of 173 consecutive head injury events with the potential to cause concussion were reported during 727 matches (29 080 player hours) over the study period. All 173 players were assessed with the PSCA tool but consent for inclusion of data was not obtained in eight cases, leaving 165 players for inclusion in complete case analyses. There were no missing data on index test parameters or reference standard results. Figure 2 presents a flow chart describing the derivation of study participants.
A wide range of mechanisms of injury were observed with head contact during tackles predominating (n=68, 41.2%, either being tackled or tackling). PSCA tests were most commonly requested by, and performed by, team doctors (request: n=145, 87.8%; performed: n=117, 70.9%). A range of support tools were used to confirm or refute the final diagnosis of concussion, of which the Sport Concussion Assessment Tool 2nd Edition (SCAT2) was the most common (n=87, 52.7%). Sample characteristics are shown in table 1.
Primary analyses: accuracy of the PSCA tool
Sixty-five of the included players had a confirmed postmatch clinical diagnosis of concussion, giving a target disorder prevalence of 39% (95% CI 32.0% to 47.3%). Of these, 55 had a positive PSCA test (true positives) resulting in a sensitivity of 84.6% (95% CI 73.5% to 92.4%) for identifying concussion. One hundred players were reference standard negative with no confirmed concussion, of which 74 cases were classified as true negatives with negative PSCA test results. The specificity to correctly identify players without concussion in this study group was therefore 74% (95% CI 64.3% to 82.3%). The positive and negative predictive values of the PSCA tool were 67.9% (95% CI 56.6% to 77.8%) and 88.1% (79.2% to 94.1%), respectively. Table 2 summarises the performance of the PSCA tool in a 2×2 contingency table. Table 3 presents point estimates of metrics of test accuracy together with their precision. The characteristics of true-positive and false-negative cases are shown in web appendix 2. There were no obvious distinguishing features of false-negative cases.
The separate subcomponents of the PSCA tool notably varied in their sensitivities and specificities. No individual subscale demonstrated satisfactory sensitivity alone, with clinical symptoms demonstrating the highest sensitivity for identifying concussion (sensitivity 76.9%, 95% CI 64.8% to 86.5%). In contrast, clinical signs, Maddocks’ questions and the tandem balance test all manifested high specificity (specificity point estimates 93–95%). The sensitivity and specificity of individual components of the PSCA tool are detailed in web appendix 2.
Secondary analyses: implementation of the PSCA tool
Decisions on exclusion from play deviated from that indicated by the PSCA tool result in 22 (13.3%) cases. Conspicuously, four cases that were ultimately confirmed with concussion had positive findings on subcomponents of the PSCA tool (abnormal tandem balance test: 1; positive symptoms: 3) but returned to play despite their positive PSCA. Conversely, three cases had no documented abnormal finding on any of the PSCA subscales, with removal from the game presumably enforced due to clinical suspicion. There were no significant differences between match-day doctors and team physicians (Pearson's χ2 test, p=0.47) in return to play decisions for patients with positive PSCA test results.
The sensitivity of actual removal from play decisions for identifying concussion was 83.1% (95% CI 71.7% to 91.2%), with a specificity of 77% (95% CI 67.5% to 84.8%). The positive and negative predictive values for the observed implementation of the PSCA tool were 70.1% (95% CI 58.6% to 80.0%) and 87.5% (95% CI 78.7% to 93.6%), respectively. In the contingency that removal from play had been enforced with a positive PSCA result, but clinical judgement could override a negative PSCA result, the sensitivity of the PSCA tool would have improved to 89.2% (95% CI 79.1% to 95.6%) with a corresponding reduction in specificity to 68% (95% CI 57.9% to 77.0%).
Web appendix 2 provides further details on observed decisions on removal from play compared with the actual PSCA test results, the accuracy of PSCA test implementation and the performance of the index test under different assumptions for clinician compliance with the PSCA tool.
Summary of results
The PSCA tool demonstrated a sensitivity of 85% and a specificity of 74% for the ultimate presence of clinically diagnosed concussion in elite rugby players. Given the reported prevalence, team doctors could be between 79% and 94% sure that a player will not have concussion following a negative PCSA test at a 95% confidence level. False-negative cases appeared to be similar to true-positive patients with no notable differing characteristics. Individual components of the PCSA tool did not have sufficient sensitivity to be used individually for the identification of concussion. There were no major barriers identified to evaluate the PSCA tool further in a large-scale study.
Interpretation of findings
An important consideration in determining the utility of the PSCA tool is the risk tolerance for failing to remove a suspected concussion case from play versus the repercussions of excluding a non-injured player. As there are serious concerns regarding the consequences of mild head injuries, a sufficient negative predictive value will be required, necessitating a high test sensitivity given the large observed prevalence of confirmed concussion in players presenting with the potential for concussion. However, given the potential detriment to team performance from excluding non-concussed players, acceptable test specificity is also important to maintain stakeholder credibility in the PSCA tool.
Although the PSCA tool exhibits potentially favourable test characteristics, instrument subcomponents were individually inadequate, confirming the need for a multimodal assessment of suspected concussion. The very low sensitivity manifested by the Maddocks’ questions alone challenges the common practice of using this test as a single screening assessment.2 Conversely, the relatively higher sensitivity of assessing detailed concussive symptoms may highlight the need for off-pitch assessments.
Clinicians’ decisions on players’ removal from play deviated from that indicated by the PSCA tool in a substantial minority of cases. The observed lack of compliance with positive PSCA results may reflect lack of training or awareness of the study protocol. Alternatively, in the real world, practice clinicians may have had greater confidence in their own experience and personal clinical acumen than in an experimental assessment tool. Pressure to return players to competitive play did not appear to be a factor in decisions, with no significant difference in non-compliance with positive PSCA results observed between PSCA implemented by team and match-day doctors. Interestingly, there was the potential for improved negative predictive value if negative PSCA test results were overruled on clinical grounds (as intended in the PSCA process), but positive results were fully implemented.
There are other important practical considerations relevant to the PSCA instrument, in addition to narrow attention to test accuracy. The ease of administration, influence on outcome of matches and acceptability to relevant stakeholders will be of great importance in any implementation. The IRB PSCA procedure is consistent with recommendations from the American Medical Society for Sports Medicine,1 and broadly follows the Zurich consensus statement guidance.8 However, after a head impact event, the Zurich approach advocates an undefined appraisal of symptoms and signs, followed by a 15 min break for players with suspected concussion, prior to a formal concussion assessment. In contrast, the PSCA process immediately and permanently removes players with obvious signs of concussion from play, and objectively screens other players for suspected concussion with the PSCA tool. The opportunity to return to play after the 5 min assessment period, if suspected concussion is excluded, combined with the PSCA tool performance reported herein, is likely to have increased credibility and applicability to competitive rugby.
The optimal timing for the assessment of suspected concussion has not been fully defined and false-negative cases could arise if players who initially appear normal develop a delayed onset of symptoms.3 It has been reported that up to 16% of players with evidence of concussion at 3 h postinjury are originally asymptomatic, some of whom may be detected with later pitchside assessment.18 A period of rest before testing could also alleviate the potentially confounding influence of fatigue on a player's test performance.8 However, the natural history of concussive symptoms is uncertain, and other suspected concussion assessment instruments have demonstrated superior sensitivity with immediate testing,19 supporting the PSCA tool's early application.
Comparison with related studies
Recent systematic literature reviews have highlighted the paucity of studies evaluating the accuracy of pitchside tests for identifying concussion during adult sport.1 ,3 Identified studies reporting measures of test performance are summarised in table 4.16 ,19–21 Although these investigations have been limited to case–control studies, which are well known to strongly overestimate test performance,22 the PSCA test demonstrates comparable or superior sensitivity. The external validity of these previous studies is also questionable given the non-rugby study samples. Additionally, many established concussion assessment tools have not been validated in the pitchside setting, for example, the NFL Sideline Concussion Assessment Tool.23
There are no other comparable non-concussive pitchside tests with which to compare the PSCA tool's performance. Clinical decision rules used in other injuries, including the Ottawa ankle rule or Ottawa knee rule,24 ,25 are derived for use in hospital environments to determine imaging decisions, and are calibrated to 100% sensitivity. Such tests have demonstrated higher sensitivity than the PSCA test (97.6% and 95.5%, respectively), but at a much lower specificity level (31.5% and 48.6%), which is likely to be unacceptable in professional rugby.24 ,25
Strengths and limitations
There are several strengths to this study. The index tests and reference standard were independently applied with no potential for partial verification bias. Additionally, the broad source population should ensure that these results are robust to spectrum effects and are generalisable throughout elite rugby competitions in developed countries.
However, there are limitations which may challenge internal validity. First, and most importantly, assessment of the reference standard was not blinded to the index test result, with both tests potentially performed by the same clinician. Diagnostic review bias may have resulted and, given the subjective nature of concussion assessment, knowledge of the PSCA result could have influenced the final clinical diagnosis.
Second, selection bias may have arisen from incomplete enrolment or study dropouts. The average number of PSCA tool tests varied across the studied tournaments, which may suggest that not all players potentially eligible for the PSCA process were identified. Moreover, a small number of players were excluded due to unobtainable consent. Systematic differences between study participants and excluded players could have biased accuracy estimates. Any ensuing selection bias could be ameliorated through video surveillance of matches and improved consent procedures in a future definitive study.
Third, the reference standard was not standardised for all cases, with variations in timing of the final assessment and in choice of the concussion support instrument used, resulting in a consequent risk of differential verification bias. If there is correlation between the performance of a particular reference standard and certain test results, measures of accuracy could be affected unpredictably. Additionally, even though the final diagnosis of concussion was based on clinical judgement, similarity between assessment domains of the PSCA tool and postmatch concussion support instruments could conceivably lead to incorporation bias.
Fourth, as acknowledged in the Zurich consensus document, the diagnosis and definition of concussion is an evolving area of medicine.8 Misclassification of the reference standard by inaccurate clinical assessment could therefore lead to errors in the reported accuracy metrics. Players may also have deliberately concealed symptoms to avoid missing games through graduated return to play protocols.26 Furthermore, reference standard misclassification could also have arisen from temporal changes in players’ clinical condition, with some false-positive cases sustaining short-lived concussions that had resolved by the time of the reference standard assessment, although the clinical significance of such injuries is uncertain.
Fifth, test reproducibility has not been assessed, and unreliability could adversely affect the accuracy and clinical utility of the PSCA instrument. Ideally, the intraobserver and interobserver reliability of administering the index test would be assessed in any future study (eg, using the κ statistic), although the practicalities of independent clinicians implementing the PSCA tool within a 5 min assessment window during a professional sports match may preclude this investigation.
Sixth, this was a pilot study aiming to test the feasibility of evaluating the PSCA tool in a future conclusive study, rather than to provide precise estimates of sensitivity and specificity. The sample size is relatively small and the reported 95% CIs are consistent with either favourable or unfavourable performance metrics for the PSCA tool.
Finally, it is possible that individual players sustained multiple head impact events, within or between games, with clustering of outcomes. Owing to the anonymised nature of the data collection, it was not possible to account for this. Although the intracluster correlation might be expected to be minimal, the reported CIs may not have the exactly correct coverage.27
This pilot study has provided the first preliminary estimates for the performance of the IRB PSCA tool, suggesting a potentially favourable balance between positive and negative predictive values. The reported results have also offered a strong basis to design and conduct a further larger trial, providing information for sample size calculations and highlighting areas for methodological development.
The PSCA working group will now proceed to a definitive study to provide a precise estimate of the PSCA tool's accuracy and to evaluate the PSCA process in detail. Study design improvements will include compulsory PSCA training for match-day medical staff, standardised reference standard assessment postgame and at 48 h, and a video review system to identify all suspicious head injuries. Final decisions on the utility of the PSCA process will ultimately depend on a judgement of its observed performance against the perceived risks of missed concussions.
What are the new findings?
Implementation of the International Rugby Board Pitch Side Concussion Assessment (IRB PSCA) tool in elite rugby appears feasible.
The PSCA tool demonstrated a sensitivity of 84.6% and a specificity of 74% in the identification of players subsequently diagnosed with confirmed concussion.
There were no major barriers identified that would prevent the evaluation of the PSCA process or tool in a future large-scale study.
How might it impact on clinical practice in the future?
Accurate pitchside assessment of players sustaining head injury events will reduce the number of players returning to play with subsequently confirmed concussion.
The authors gratefully acknowledge the support and input of Dr Martin Raftery (IRB Chief Medical Officer) in the development of the PSCA Tool and Process. The authors also thank the following members of the IRB PSCA working group for their expert recommendations: Michael Makdissi (The Florey Institute of Neuroscience and Mental Health, Australia), Conor McCarthy (Irish RFU Medical Director), Deborah Robinson (New Zealand RU Medical Director), Rob Nichol (Chief executive International Rugby Players' Association), Vincenzo Ieracitano (Chairman Medical Commission of Italian FIR), Mike England (Director English RFU Injured Players Foundation), Paul Watson (Medical Director Rugby Canada – independent advisors); Mark Harrington (IRB Training and Medical Manager – IRB representative). Methodological advice from Matt Cross and Keith Stokes (University of Bath), and data management by Marc Douglas (IRB Research Coordinator), are also gratefully acknowledged.
Correction notice This paper has been amended since it was published Online First. The third author's affiliation has been changed and the wording ‘French Federation of Rugby’ has been added.
Contributors PD and SPTK conceived and designed the study in collaboration with the International Rugby Board PSCA working group. They also critically revised the manuscript for important intellectual content and gave final approval to the version to be published. GWF analysed and interpreted the data, and wrote and prepared the manuscript for publication.
Competing interests None.
Ethics approval The study protocol received ethical approval from the University of Bath for the Aviva Premiership.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Further details on unpublished data are available on request by contacting the corresponding author, subject to permission from the International Rugby Board.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.