Background/aim Despite concussion education being increasingly mandated by states and sports leagues, there has been limited evaluation of what education is in fact effective. The National Collegiate Athletic Association (NCAA) currently mandates that institutions provide concussion education, without specifying content or delivery. The present study evaluated the effectiveness of this general mandate, as enacted for male collegiate ice hockey teams within one conference of competition.
Methods In a prospective cohort design, 146 players from 6 male collegiate ice hockey teams in one Division 1 conference completed written surveys before and after receiving their institution-determined concussion education. Knowledge, attitudes, perceived norms and behavioural intention were assessed using validated measures. Education content and delivery was assessed by open-ended responses and consultation with team athletic trainers.
Results All teams received concussion education material; however, content and delivery varied. Rates of material recall differed by delivery format. Considering all teams together, there were no significant improvements in knowledge and only a very small decrease in intention to continue playing while experiencing symptoms of a concussion. Pre-education and post-education, there were significant between-team differences in attitudes towards concussion reporting and behavioural intention.
Conclusions The NCAA's general education mandate was divergently enacted; it did not significantly change the constructs of interest nor did it mitigate the pre-education team differences in these constructs. Existing educational materials should be evaluated, theory and evidence-driven materials developed, and mandates extended to, at a minimum, recommend materials found to be effective in changing concussion-reporting behaviour.
- Ice hockey
Statistics from Altmetric.com
A growing body of scientific evidence links concussions and other repetitive brain trauma to short-term and long-term neurological deficits. In the acute phase, concussions have been linked to a wide array of symptoms including the following: difficulty concentrating or remembering, sleep disturbances, cognitive deficits, irritability, depression, and suicidal thoughts and behaviours.1 In addition to immediate deficits, brain trauma can also result in long-term problems including behavioural issues,1 ,2 cognitive issues3 and potentially Chronic Traumatic Encephalopathy,4 a neurodegenerative disease that presents with changes in cognition, mood and behaviour.4–8 This disease has been found in a variety of individuals, including those with a history of participation in professional and/or amateur contact sports.4 ,5 After exposure to brain trauma, even when a concussion is not formally recognised, athletes can have changes in brain white matter as measured by diffusion tensor imaging,9 functional impairments as measured by functional MRI and neuropsychological examination,10–12 and changes in the blood brain barrier.13 Exposure to additional brain trauma while recovering from the initial impact has been associated with magnified neurological consequences14–20 and controversially linked to Second Impact Syndrome, a condition that can result in long-term neurological deficits and sometimes death.21–23 Continued play or premature return to play after a concussion is a dangerous problem that has been identified in multiple sporting populations.24–27 Some evidence suggests that reporting behaviours may be associated with an athlete's knowledge about concussions.12 ,28–30 However, recent focus groups conducted with male and female high school athletes suggest that rather than concussions knowledge, factors in the athletic environment, such as coach approachability and perceived norms, may be the key factors affecting concussion reporting.31
In part to increase volitional player-report of suspected concussions to coaching or medical personnel, a vast majority of states have adopted concussion legislation for youth athletes, often including mandates about the provision of concussion information.32 Of primary interest to the present study, the National Collegiate Athletic Association (NCAA) policy states as follows: ‘Student-athletes should be provided with educational material on concussions’.33 Currently, there are no requirements about the content or delivery of this education and no evaluation of the type of materials delivered to players under this mandate, how they are delivered, and whether they are effective in changing reporting-relevant cognitions.
Evaluating education effectiveness in changing reporting-relevant cognitions requires the need to specify what the relevant cognitive constructs may be. Existing concussion education programmes for athletes have tended to focus on symptom identification and reporting protocol,34–36 with evaluation largely assessing change in concussion knowledge. Given that a goal of concussion education is to change player behaviour (eg, ceasing play when symptoms are present), concussion knowledge may not be the best or only factor to target. Provvidenza et al37 have outlined the importance of using appropriate knowledge transfer principles in the development and evaluation of effective concussion education, including assessing context-specific knowledge needs, attitudes, barriers to knowledge use and delivery strategies. The broader field of injury prevention called nearly a decade ago for the explicit application of behaviour-change theory to injury prevention programmes.38 ,39 However, a recent systematic review of the extent to which behavioural and social science theories are used in sports injury prevention research finds that only 11% of studies, and no concussion education programmes for athletes, explicitly include theory in programme design, implementation or evaluation.40 Chrisman et al31 suggest that the Theory of Planned Behavior (TPB)41 may be an appropriate frame through which concussion reporting behaviours may be understood, with emphasis on the potential importance of perceived reporting norms and attitudes towards reporting. This expectancy value theory presumes that an individual's attitude towards the behaviour, perceived behavioural norms for important referents, and belief about their ability to perform the behaviour, all predict behavioural intention, which in turn predicts performance of the behaviour.41 Within the formulation of TPB, Ajzen and Fishbein41 propose that knowledge is predictive of behaviour only to the extent it informs subjective norms or links the behaviour to salient personal consequences. Of the few sports injury prevention programmes that have explicitly incorporated theory in their design, implementation or evaluation, TPB is among the theories most frequently used.40
Informed by TPB and the importance of using context-appropriate knowledge transfer strategies,34 the aim of the present study was to assess the education provided to one conference of male Division 1 hockey players in terms of content, delivery and effectiveness in changing theory-driven concussion-reporting relevant cognitions. The present study also aimed to assess previous season post-head impact reporting behaviour and reasons for non-report of suspected concussions, behavioural data not yet available about male collegiate ice hockey players.
Teams were recruited by contacting coaches from one NCAA Division 1 men's hockey conference. Coaches of six league teams agreed to allow their team members to participate on a voluntary basis, with coaches of the remaining teams declining participation. Compared with the teams that participated, the teams that declined participation had similar win rates over the previous season (p=0.103). Individual player participation was voluntary and subject to informed consent (n=146, 93% participation). Participant characteristics by team are summarised in table 1. All research activities were approved by the Harvard School of Public Health Institutional Review Board.
A prospective cohort design was used to assess the content and delivery of respective concussion education provided to teams, and the overall effectiveness and between-team variability in effectiveness of the NCAA's mandated concussion education. Two surveys were administered to teams at the beginning of the 2012–2013 academic year, prior to the start of the hockey season, with survey completion generally being the day before and the day after institution-delivered concussion education. Team 3 was the one exception to this procedure. The team's athletic trainer reported delivering educational materials to the team by email over the summer, prior to the team's return to campus. Consequently, only survey 2 was administered to this team. Surveys were administered directly to each team by research staff and completed by hand in a quiet location. The content of the team's preseason concussion education was reported to the research staff by the team's athletic trainer.
Surveys assessed concussion knowledge, reporting attitudes, perceived reporting norms and reporting intention using scales and indices drawn from Rosenbaum & Arnett's Rosenbaum Concussion Knowledge and Attitude Scale.42 Knowledge was measured using Rosenbaum & Arnett's 25-question Concussion Knowledge Index (CKI), part of the Rosenbaum Concussion Knowledge and Attitudes Survey—Student Version.42 CKI consists of 25 true/false items assessing basic concussion knowledge and has been previously validated in a population of adolescent athletes.43 Attitudes and subjective norms were measured using a modified version of Rosenbaum & Arnett's Concussion Attitudes Index (CAI). Validation of CAI suggests a multiple-factor structure, with factors including personal concussion attitudes and others’ concussion attitudes.42 For the present study, items from CAI pertaining to the individual's own attitude towards concussions were considered part of the Attitudes measure, while items pertaining to perceptions of others’ attitudes towards concussion were considered part of the Perceived Norms measures. Items were scored on a five-point scale, ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). Attitudes contained 10 items for a maximum score of 50 while Perceived Norms contained five items for a maximum score of 25. Internal consistency reliability for both scales was adequate (Cronbach's α=0.83 and 0.81, respectively). Behavioural intention was measured using Rosenbaum & Arnett's item “I would continue playing my sport while experiencing a headache that results from a minor concussion,” similarly scored on a five-point Likert scale.42
Previous-season incidence of concussion, sensory, somatic and cognitive symptoms and postinjury reporting behaviour were assessed using Kaut et al's28 Head Injury Questionnaire. Among student-athletes who during the previous-season did not report an impact that they believed was a concussion, reasons for not reporting were assessed with categories previously used by McCrea et al.24
Open-ended written responses asked players to describe the concussion education received from their institution this year. Players were asked to describe how effective they found this education, and what education content and delivery format they believed would be most effective.
Framed by a positivist paradigm, basic qualitative methods were used to code the open-ended participants’ responses.44 Responses about current education were coded using structural coding and frequency counts informed by athletic trainer-reported education. Desired education was analysed inductively using content analysis to generate categories. An independent second coder subsequently applied these categories to the data, achieving 100% inter-rater reliability.
For all between-team comparisons, a one-way analysis of variance was used to identify differences for continuous variables, with the Sidak test used to assess post hoc pairwise comparisons. For categorical variables, a χ2 test was performed to identify differences between teams, with Fisher's exact test being used when necessary due to a small sample. For all outcome measures of interest (knowledge, attitudes, perceived norms and behavioural intention), the Bonferroni correction was used to account for multiple comparisons. Multivariate linear regression was performed for variables of interest using a forward stepwise model selection paradigm to create a parsimonious model. All analyses were performed with IBM SPSS V.20.
While all sample institutions adhered to the NCAA's general mandate and provided concussion education to the men's ice hockey teams in some capacity, this education differed by institution in terms of content and delivery. The type of educational material (handout, email, lecture from trainer and/or video) is reported for each team in table 2, along with the percentage of team members who reported remembering that they had received each type of educational material the following day. All teams received written materials about concussions, with nearly all teams receiving the CDC/NCAA Concussion Fact Sheet for Student Athletes publicly available on the Center for Disease Control and Prevention's website.45 However, the delivery of this symptom-focused information varied between teams, as did the percentage of team members recalling that they had received the handout. Members of two teams received this information via email (recall rates of 0% and 26%), members of three teams received their own hard copy of the handout (recall rates of 31%, 74% and 76%), and members of one team had the handout left in the locker room for common perusal (recall rate of 26%). Five of the six teams received a lecture from their athletic trainer about concussions (recall rates ranging from 28% to 83%). Only one team also received information in video-form, this form of delivery having the highest rate of recall across all teams and types of educational materials (92%). The video, a joint production of the National Academy of Neuropsychology Foundation, the National Athletic Trainers’ Association and the NHL Players’ Association,46 was targeted specifically at hockey players, with information provided by current and former professional hockey players as well as doctors. In response to a prompt asking what changes would make their team's concussion education more effective, players on other teams most frequently reported wanting information delivered in video form, with testimonials from former athletes about the long-term consequences of concussions.
Results of education effectiveness are reported in table 3. To assess the effect of education in changing reporting-relevant cognitions, changes were first assessed for the sample as a whole. No statistically significant changes were observed in knowledge, attitudes or perceived norms (p=0.38, 0.78 and 0.11, respectively). A small change was observed in behavioural intention (–0.25, p=0.004). Although statistically significant, the effect size (Cohen's d=0.121) falls below Cohen's guideline of 0.20 for a small effect.47 Athletes who correctly recalled the education received were compared with athletes who incorrectly recalled this education in terms of change in reporting-relevant cognitions. Change in intention to continue playing with a minor concussion was significantly higher among athletes who reported remembering that they had received a lecture (p=0.021) and significantly lower among those who remembered that they had received an email (p=0.017), with no significant differences for other permutations of delivery format or for correct recall of any format. An athlete's baseline concussion knowledge had no influence on whether or not they accurately remembered the type of education they received.
Next, between-team variability in change scores for each cognitive measure was assessed. Significant between-team differences were only observed for knowledge change (p=0.041), with the team that watched the video (Team 6) having the greatest improvement in knowledge (+0.73), and two teams (Teams 1 and 4) actually scoring worse on the measure of knowledge following education (−0.426 and −0.227, respectively). Given the limited changes in cognitions as a result of the education provided, posteducation between-team differences were assessed to determine whether there was between-team variability not attributable to institutional concussion education. Significant between-team differences were found in posteducation attitudes (p=0.010) and posteducation behavioural intention (p=0.008), with video-watching Team 6 having the lowest intention to continue playing with a minor concussion.
Given the relatively minimal effect of education on changing reporting-related cognitions, or explaining between-team variability in intention to continue playing with a minor concussion, multivariate linear regression was conducted to determine which variables had a significant influence on change in behavioural intention and posteducation behavioural intention. Behavioural intent at follow-up could be predicted from knowledge at baseline (β=−0.164, p=0.012), knowledge at follow-up (β=0.269, p<0.001), attitudes at follow-up (β=−0.168, p<0.001), perceived norms at follow-up (β=0.229, p<0.001), and whether an athlete had been diagnosed with a concussion in the previous season (β=−0.474, p=0.026) with R2=0.647. Change in behavioural intent could be predicted from knowledge at follow-up (β=0.118, p=0.043), attitudes at baseline (β=0.100, p=0.002), attitudes at follow-up (β=−0.128, p<0.001), perceived norms at baseline (β=−0.113, p=0.041), perceived norms at follow-up (β=0.130, p=0.025), and whether an athlete had been diagnosed with a concussion in the previous season (β=−0.527, p=0.029) with R2=0.305.
As reported in table 1, during the previous (2011–2012) ice hockey season, 18.5% of participants reported being diagnosed with at least one concussion by a medical professional, with significant variability between teams (0–33.3%). Many more participants reported experiencing symptoms after a head impact that could be suggestive of a concussion. In total, 45.9% of participants reported playing with a headache after being hit on the head, with significant variability between teams (12.5–54.2%). As reported in table 4, among athletes who suspected that they had a concussion during the previous season but did not report their injury, 50.6% stated that they did not report it because they did not know it was a concussion, 69.7% did not think it was serious enough, 48.3% did not want to be pulled out of the game or practice and 32.6% did not want to let down their teammates.
Concussions from contact sport are becoming increasingly recognised as a significant public health problem,48 ,49 with scientific evidence pointing to immediate as well as long-term neurological and functional consequences. Evidence suggests that athletes often continue to play after sustaining a concussive impact,25 and this appears to be the case among participants in the present study. As part of a strategy to reduce the burden of concussions attributable to under-reporting, states and sports organisations have increasingly mandated concussion education for players. In particular, the NCAA requires that student-athletes be provided with ‘educational material about concussions’.33 While this is a laudable first step, it is important to understand whether this directive results in concussion education that is effective in changing player symptom-reporting cognitions and behaviours.
The sample as a whole shows no change in knowledge or attitudes but a small but statistically significant improvement in intention to continue playing with a minor concussion following concussion education; however, this effect size is of little practical significance. The ineffectiveness of the existing educational materials in changing relevant cognitions, and the low rate of recall for many modes of delivery, is not surprising given the didactic focus on symptom identification and reporting. Provvidenza et al37 outline the importance of choosing modes of information delivery that are context appropriate and suggest that various forms of electronic media hold promise for evaluation. Within the present sample, the video watched by Team 6 appeared to be the most effective form of education, even if only by the metric that team members consistently remembered viewing it, unlike materials delivered in other formats. Provvidenza et al37 also outline the importance of identifying the knowledge needs of the target population and adapting the content of educational materials to those specific needs. Presently, concussion education efforts—including the education delivered to the teams in this study—tend to focus on symptom knowledge, which may be constraining education effectiveness in changing reporting intention and behaviour. The importance of targeting cognitions other than just concussion identification is reinforced by the largely volitional reasons given by participants in the present sample who believe that they sustained a concussion during the previous season but did not report it. Chrisman et al31 suggest that the perceived consequences of reporting subjective reporting norms may be important predictors of concussion reporting behaviour and that they may be important constructs for concussion education programmes to target. The present findings of a significant association among knowledge, attitude and subjective norms and concussion reporting intention reinforce the importance of these cognitions as targets of concussion education and the potential appropriateness of TPB as a theory to inform the design and evaluation of programmes attempting to modify concussion reporting behaviour.
At a team level, while there were significant between-team differences in change in knowledge, there were no significant differences in change in attitudes, norms or behavioural intention attributable to the education. This is not unexpected as the education provided to teams focused on symptom knowledge and reporting protocol and the delivery (and recall) varied by team. We do find that posteducation behavioural intention varies significantly between teams, with Team 6's score of 1.82/5 being notably lower than that of all other teams. Despite the fact that Team 6 appeared to have the most ‘successful’ education—with nearly all members of the team at least recalling that they watched a concussion education video, and with knowledge increasing significantly more than other sample teams—intention to play with a concussion was lower than other sample teams pre-education and did not decrease significantly more than other teams. Further, during the previous hockey season, members of Team 6 also had the greatest percentage of team members diagnosed with at least one concussion (33.3%) and the lowest percentage reporting that they continued to play with a headache after being hit in the head (12.5%). While there does not appear to be a direct link between the concussion education (including video) provided to Team 6 during the study period and the team's reporting intention, a third variable may explain the apparent association between the more sophisticated and well-received concussion education strategy and the team's less dangerous reporting-related cognitions. One possibility is that the athletic training and/or coaching staff of Team 6 are more aware of the importance of effectively communicating with players about concussions and may, as a result, choose concussion education materials that reflect more than the bare minimum of passing out a symptom sheet and saying a few words. Independent of the effect of the officially mandated education, they may also foster a culture of safety and reporting through other formal and/or informal means. Future education evaluation efforts are encouraged to simultaneously assess coach and team medical personnel knowledge, attitudes and communication with athletes about concussions and determine how factors related to coach and medical staff covary with education content, delivery and effectiveness.
Despite the apparent ineffectiveness of the NCAA's current general concussion education mandate, the present findings should be interpreted with caution, particularly in the light of the relatively modest sample size. Given the low rate of recall for many of the types of educational materials, we should highlight that our study assessed the effectiveness of education, as delivered in-context, and not necessarily the efficacy of the educational materials themselves. Setting variant effectiveness research has been described as an important component of programme evaluation50 and is particularly relevant for the present study, given the uncontrolled delivery of information from athletic trainers to teams. Additionally, while the very high individual response rate eliminates the possibility of response bias within-teams, not all league teams participated, with team coaches making the final decision about team eligibility. Despite the similarity of participating and non-participating teams in terms of win percentages over the previous season, it is very possible that coach attitudes about concussions and concussion education are systematically related to team participation. Future research efforts are encouraged to form partnerships with sports leagues and conferences to facilitate higher rates of team participation. Further, the generalisability of these findings to other sports, ages, genders and levels of competition may be limited. Future research with larger samples is encouraged to evaluate the content and effectiveness of existing concussion education in other sporting populations. Finally, while observational and experimental evidence relatively consistently finds that behavioural intention predicts behaviour,51 ,52 this has yet to be tested for concussion reporting. We suggest that future evaluation efforts include longer term behavioural follow-up and test whether behavioural intention posteducation predicts subsequent reporting behaviour.
The NCAA was founded over 100 years ago to protect the health of student-athletes. While it should be commended for beginning to address the issue of concussions by mandating education, and recently creating the position of Chief Medical Officer, our findings suggest that more must be done if the health of student-athletes is to be adequately protected. Among the present sample of Division 1 hockey players, a general directive to provide concussion education to players does not appear to be effective. At a minimum, we recommend that the NCAA's directive be expanded to recommend concussion education materials that have demonstrated effectiveness in increasing concussion reporting intention and behaviour. Beyond the NCAA, as states and other sports leagues mandate concussion education for players, it is important that theory-driven education programmes are developed and evaluated, and that policy recommendations with respect to education programmes are evidence based.
What are the new findings?
The general National Collegiate Athletic Association concussion education mandate results in between-team variation in education content and delivery.
The education provided does not significantly change concussion-reporting cognitions.
There is a high rate of concussion under-reporting among male college hockey players, and significant between-team variation in attitudes and intentions related to concussion reporting.
How might it impact on clinical practice in the near future?
Concussion education provided to athletes by clinicians and institutions should be theory-driven and evaluated for effectiveness.
Addressing only concussion knowledge is not sufficient; cognitions such as attitudes, perceived norms and intentions are also important targets.
Contributors All authors made substantial contributions to (1) the conception and design or acquisition of data, or analysis and interpretation of data; (2) drafting the article or revising it critically for important intellectual content and (3) final approval of the version to be published.
Funding National Collegiate Athletic Association (NCAA) 2012–2013 Graduate Student Research Grant.
Competing interests None.
Ethics approval Harvard University School of Public Health.
Provenance and peer review Not commissioned; externally peer reviewed.
▸ References to this paper are available online at http://bjsm.bmj.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.