Article Text

PDF

Computerised cognitive assessment of concussed Australian Rules footballers
  1. M Makdissi1,
  2. A Collie2,
  3. P Maruff2,
  4. D G Darby3,
  5. A Bush4,
  6. P McCrory1,
  7. K Bennell1
  1. 1Centre for Sports Medicine Research Education, University of Melbourne, Parkville, Victoria, Australia
  2. 2Neuropsychology Laboratory, Mental Health Research Institute of Victoria, Parkville, Victoria, Australia
  3. 3Centre for Neuroscience, University of Melbourne
  4. 4Behavioural Neurology Laboratory, Mental Health Research Institute of Victoria
  1. Correspondence to: Dr Collie, Neuropsychology Laboratory, Mental Health Research Institute of Victoria, Locked Bag 11, Parkville, Victoria, 3052, Australia alex{at}neuro.mhri.edu.au

Abstract

Background—“Paper and pencil” neuropsychological tests play an important role in the management of sports related concussions. They provide objective information on the athlete's cognitive function and thus facilitate decisions on safe return to sport. It has been proposed that computerised cognitive tests have many advantages over such conventional tests, but their role in this domain is yet to be established.

Objectives—To measure cognitive impairment after concussion in a case series of concussed Australian Rules footballers, using both computerised and paper and pencil neuropsychological tests. To investigate the role of computerised cognitive tests in the assessment and follow up of sports related concussions.

Methods—Baseline measures on the Digit Symbol Substitution Test (DSST), Trail Making Test-Part B (TMT), and a simple reaction time (SRT) test from a computerised cognitive test battery (CogState) were obtained in 240 players. Tests were repeated in players who had sustained a concussive injury. A group of non-injured players were used as matched controls.

Results—Six concussions were observed over a period of nine weeks. At the follow up, DSST and TMT scores did not significantly differ from baseline scores in both control and concussed groups. However, analysis of the SRT data showed an increase in response variability and latency after concussion in the injured athletes. This was in contrast with a decrease in response variability and no change in latency on follow up of the control players (p<0.02).

Conclusion—Increased variability in response time may be an important cognitive deficit after concussion. This has implications for consistency of an athlete's performance after injury, as well as for tests used in clinical assessment and follow up of head injuries.

Statistics from Altmetric.com

Take home message

This paper illustrates the practical problems of using “pen and paper” sideline neuropsychological testing for sport related concussion. Computerised test batteries, such as CogState, both allow a wider range of domains to be tested and have the ability to detect fatiguability or variability in cognitive performance, which in turn is a key measure of recovery.

Head injuries are common in many sport and recreational activities, particularly those involving contact—for example, football and boxing—and high speeds—for example, skiing and motor car racing. Although most injured athletes appear to recover uneventfully from a single concussive episode, it has been proposed that repetitive mild head trauma may be implicated in the development of diffuse cerebral swelling (the so called “second impact syndrome”)1,2 and cumulative cognitive deterioration,3, 4 although this remains controversial. Thus, a key issue in the management of sports related concussion is determining when it is safe to allow the athlete to resume participation after injury.

The absence of validated criteria for assessment of injury severity complicates decisions about the timing of safe return to sport. Such decisions are currently made with reference to the presence and severity of symptoms after concussion. However, such symptoms can be variable5 and typically resolve before changes in cognitive function have recovered.6 Symptoms may also be underrated by an athlete who is keen to return to sport. Neuropsychological tests are now commonly used to provide an objective measure of cognitive function and recovery after a concussive injury. Given that individuals vary considerably in their performance on many neuropsychological tests, interpretation of results after concussion is facilitated by a knowledge of baseline scores for each player.

Using such comparisons with baseline, recent studies have reported impairments on tests of information processing and psychomotor speed in head injured athletes.3–6 For example, the Digit Symbol Substitution Test (DSST), derived from the Weschler Adult Intelligence Scale-Revised,7 has been shown to be a sensitive and robust measure of cognitive function after concussion in Australian Rules football.8 Several equivalent forms of the test exist, which may minimise the practice effects that occur with repeated test administration. Other tests such as the Trail Making Test-Part B (TMT) and the Paced Auditory Serial Addition Test (PASAT) have also been used; however, these tests do not consistently reveal deficits in cognitive function after concussion.6, 9 Although studies such as these provide useful information for the sports physician, they suffer from a number of limitations.10 Ideally, the instrument used for neuropsychological assessment of the concussed athlete should be portable and have a brief administration time. These qualities may enable baseline testing to be conducted in large groups of players before the start of the season, while also facilitating sideline assessment of players after a concussive injury. Such tests have been developed recently,11 but they may not allow comprehensive evaluation of cognitive processes and may be limited with regard to sensitivity.12

It has been suggested recently that computer based cognitive tests may be more sensitive to cognitive impairment after sports related head injury than conventional neuropsychological tests.10, 12, 13 This arises partly from observations that computerised tests of simple and choice reaction time (RT) have demonstrable sensitivity for detecting cognitive changes after mild head injury.13–16 Further, we have proposed recently that the excellent psychometric properties afforded by the automation of response recording and stimulus presentation may enhance the sensitivity of computerised tests to cognitive deficits after concussion.10 Despite their potential, computerised cognitive tests have not yet been validated for use in the follow up of sports related concussion. The purpose of this study was to investigate the efficacy of a new computerised cognitive test battery (CogState) in the assessment and follow up of concussed Australian Rules footballers, and to compare results from this test with those obtained from conventional paper and pencil neuropsychological tests. We present a case series of concussions observed in athletes competing in the Australian Football League (AFL).

Methods

PARTICIPANTS

Participants were recruited from six Australian Rules football teams (four elite professional, one semiprofessional, one amateur) over a single season. The average age of these players was 20.5 (range 17–26). Six concussed players were assessed within the first nine weeks of the season. A group of seven matched non-injured players were recruited as controls. Table 1 gives the basic and clinical data recorded for the players. All players provided informed consent before baseline testing, and the study was approved by the University of Melbourne ethics and research committee.

Table 1

Basic and clinical data on six concussed Australian Rules footballers and seven controls

MATERIALS

The DSST and TMT17 were administered to assess speed of information processing, psychomotor function, and visuospatial ability. A 15 minute computerised cognitive test battery (CogState) was also administered to all participants. The CogState test battery includes measures of sustained and divided attention, learning and memory, problem solving, and decision making.18 This performance based test battery uses playing cards as stimuli, and is designed to have almost infinite equivalent alternative forms.10 CogState may be self administered, but in this study appropriately trained personnel oversaw all assessments. Only data from the Simple Reaction Time (SRT) tests of the CogState battery were considered for analysis. In this test, a single playing card was presented face down in the centre of a computer screen. Participants were required to press the spacebar as quickly as possible whenever the card was turned face up. The SRT test was presented three times throughout the 15 minute test battery, once as the first test (SRT1), once in the middle of the test battery (SRT2), and once as the last test (SRT3). Eighteen responses were collected during SRT1, and 15 during SRT2 and SRT3, providing a total of 48 responses for analysis.

PROCEDURE

A total of 240 Australian Rules footballers from the six teams completed CogState, the DSST, and the TMT before the beginning of the football season, after providing informed consent. All players completed the computerised test battery twice at these baseline assessments, to reduce the effects of practice in subsequent comparisons with baseline. Data from the second of these baseline assessments are reported in this study. Six players were concussed during the first nine weeks of the 2001 football season. Data for these players and seven matched non-injured controls are reported here.

We used the definition of concussion provided by the Congress of Neurological Surgeons: “ . . .a clinical syndrome characterised by the immediate and transient post traumatic impairment of neural function”.19 A clinical diagnosis of concussion was made by the medical practitioner of each club, and a standardised assessment form was used to record symptoms and orientation after injury. All neuropsychological tests were repeated within 72 hours of injury, and the scores were compared with the player's baseline performance. The time taken for the player to return to sport (full training or playing) was also recorded. This was based on the usual management protocols of the team doctor.

DATA ANALYSIS

For each player, the interval between baseline and follow up assessments was calculated in days. For the concussed players, the interval between the concussive episode and the cognitive assessment was also calculated (table 1). Data from the DSST and TMT were analysed in two ways. Firstly, for both tests, the performance of each concussed player at baseline and at follow up was tabulated, and qualitatively compared with the performance of the matched non-concussed control group (table 2). Secondly, data from each test were submitted to 2 × 2 repeated measures analysis of variance comparing group (concussed, non-concussed) by test (baseline, follow up). In the event of a significant interaction, post hoc t tests were used to investigate between group and/or between test differences.

Table 2

Individual and group mean data for the Digit Symbol Substitution Task (DSST) and the Trail Making Test-Part B (TMT)

On the SRT task, data were excluded from analysis if RTs were faster than 100 milliseconds (anticipation) or slower than 5000 milliseconds (max out). When an anticipation or max out occurred, this was recorded as an error, and a replacement trial was scheduled to the end of the block. For all participants, mean and standard deviation RTs and error rates were calculated for each of three SRT tasks, at both baseline and follow up assessments. Grand mean and standard deviation RTs and error rates were also calculated for each participant by combining data from all three SRT tasks (table 3). These data were then analysed in two ways. To investigate the primary hypotheses, a repeated measures analysis of variance was conducted on grouped data, with test (baseline or follow up) as the within subjects factor and group (concussed players v non-concussed players) as the between subjects factor. This analysis was conducted for mean RT, standard deviation RT, and error rate data. When significant interactions were observed, post hoc t tests were used to investigate between group and/or between test differences. Secondly, data for each concussed player were plotted and compared with data obtained from non-concussed control players, in order to investigate the consistency of any interaction or main effect observed in the analysis of variance. This qualitative analysis was conducted for mean (fig 1) and standard deviation RT data (fig 2).

Table 3

Individual and group mean data for the CogState simple reaction time task

Figure 1

Mean reaction times on the CogState simple reaction time task. Individual and group mean data from baseline and follow up assessments on the simple reaction time task are plotted. The change from baseline is given as a percentage. CP, concussed player.

Figure 2

Standard deviation reaction times on the CogState simple reaction time task. Individual and group mean data from baseline and follow up assessments on the simple reaction time task are plotted. The change from baseline is given as a percentage. CP, concussed player.

To determine the source of any increased variability in RTs resulting from concussion, we calculated percentile scores at both baseline and follow up assessments. For each participant, the RTs for all 48 valid responses were listed, and RTs falling at each decile of this distribution were determined. Group mean percentile scores were then calculated (table 4). Baseline percentile scores were then plotted with RT on the y axis and percentile score on the x axis (fig 3), and a linear regression equation was fitted to these data. The standard error of that regression equation was then calculated and used to define 95% confidence intervals at each percentile using the formula:

Table 4

Group mean percentile scores on the CogState simple reaction time task

Figure 3

Group mean percentile scores on the CogState simple reaction time task. Scores at the 10th to 90th percentile calculated at baseline (dashed lines), and 95% confidence intervals around these scores (dotted lines), are plotted for non-concussed (A) and concussed (B) players. Percentile scores calculated on follow up data are plotted over these data for both groups (solid lines).

Group mean percentile ± 1.96 × standard error

Group mean percentile scores calculated from follow up data were then plotted over these confidence intervals. Significant changes from baseline were said to have occurred when follow up percentile scores fell outside these confidence intervals (fig 3). Finally, data from the four concussed players retested on a third occasion were qualitatively analysed relative to their performance at baseline and follow up assessments.

Results

CLINICAL FEATURES

Of the six players concussed in the first nine weeks of the season, headache was the most common symptom reported (recorded in five players). Interestingly, four of these players reported that the headache began up to six hours after the concussive injury. Furthermore, in three of these players, headaches were the most persistent symptom recorded, lasting up to four days. In the remaining players, fatigue/lethargy was the longest lasting symptom. Other symptoms reported were dizziness (four), confusion (four), nausea (two), and blurred vision (one). In all players, subjective symptoms had resolved within four days of injury (range one hour to four days). Only one player (player 6) suffered loss of consciousness. This episode was brief, lasting less than one minute.

Two players were symptomatic at the time of follow up testing. Player 3 was still suffering from headaches, and player 6 had both headaches and dizziness. It is possible that the presence of these symptoms may have affected their performance on the neuropsychological tests. However, both players also had the most number of symptoms, longest time course of symptoms, and longest time taken to return to sport; therefore the performance deficits observed on the tests may reflect an increased severity of injury.

COGNITIVE TESTS

Table 2 gives the individual and group mean DSST and TMT task data. On the DSST, the performance of five of the concussed players had improved when their follow up performance was compared with baseline. On the TMT, the performance of all concussed players improved at follow up. On both tests, the magnitude of these improvements was relatively consistent between the concussed and control groups. For the DSST, analysis of variance showed a significant effect of test (F(1,12) = 10.05, p = 0.01; Eta2 = 0.51), but no effect of group (F(1,12) = 0.32, p = 0.58) and no test by group interaction (F(2,11) = 0.13, p = 0.72). Analysis of variance conducted on TMT data showed no main effects or interactions (test (F(1,12) = 1.81, p = 0.21); group (F(1,12) = 0.70, p = 0.42); test by group (F(1,11) = 0.51, p = 0.49)).

Table 3 gives the individual and group mean SRT task data. For the standard deviation data, all six concussed players displayed an increase in response variability when their performance after concussion was compared with baseline. This increase was particularly evident in players 1, 2, 3, and 6. In contrast, non-concussed control players recorded an average 36.0% decrease in response variability from baseline to follow up (fig 2). Analysis of variance showed this interaction to be significant (F(1,12) = 7.12, p = 0.02; Eta2 = 0.39). Post hoc t tests showed that the performance of concussed players was significantly more variable than control players at the follow up assessment (t(12) = 3.18, p = 0.01), but not at baseline (t(12) = −0.55, p = 0.59).

For the mean RT data, five of the concussed players displayed RT slowing after concussion. This slowing was most evident in players 1, 3, and 4, and was manifested as a group mean increase of 16.2% on baseline mean RTs. In contrast, the mean performance of non-concussed control players increased by 1.2% from baseline to follow up (fig 1). Analysis of variance conducted on these data showed a significant main effect of assessment (F(1,12) = 6.34, p = 0.03: Eta2 = 0.37) and a test by group interaction that approached significance (F(1,12 = 4.69, p = 0.053; Eta2 = 0.30). All players were highly accurate at both baseline and follow up assessments, with no significant group or individual differences. Analysis of variance conducted on the error data confirmed that there were no main effects or interactions reaching significance (test (F(1,12) = 1.89, p = 0.19); group (F(1,12) = 0.16, p = 0.62; test by group (F(1,12) = 0.22, p = 0.65)).

Table 4 gives percentile scores at baseline and follow up for both concussed and non-concussed player groups. For the non-concussed player group, a regression equation fitted to baseline data yielded a standard error of 15.0 milliseconds. Follow up scores at the 90th percentile were just outside the 95% confidence intervals derived from these data (fig 3A). For the concussed player group, a regression equation fitted to baseline data yielded a standard error of 25.2 milliseconds. Follow up scores at the 80th and 90th percentile were observed to be well outside the 95% confidence intervals derived from this data (fig 3B).

Two post hoc analyses were performed. Analysis of group data and analysis of variance results suggested a dissociation between performance on the conventional neuropsychological tests (DSST, TMT) and the computerised tests (SRT) in the concussed group, but not in the control group. Therefore we investigated the relation between changes in performance on the DSST, TMT, and SRT tests in both concussed and control groups. For each test, individual change scores were calculated by subtracting each participant's follow up score from their baseline score. These change scores were then submitted to a two tailed bivariate correlational analysis. The control group exhibited moderate correlations between mean RT and DSST (r = −0.48), mean RT and TMT (r = 0.42) and small correlations between standard deviation RT and DSST (r = −0.27), and standard deviation RT and TMT (r = 0.23). In contrast, much smaller correlations were observed between these same variables in the concussed group (mean RT/DSST (r = −0.04); mean RT/TMT (r = 0.04); standard deviation RT/DSST (r = −0.28); standard deviation RT/TMT (r = 0.08)).

Secondly, we qualitatively compared the SRT task performance of the four concussed players who had completed a third assessment between 7 and 14 days after the concussive episode with their own performance at baseline and at two to three days after concussion. Players 2, 3, 4, and 6 had such data available for analysis. For standard deviation RT, players 2 (26.8 milliseconds) and 3 (60.5 milliseconds) had performed better than at baseline, player 6 (103.7 milliseconds) had improved on his immediate post concussion score but was not yet back to baseline levels, and player 4 (200.9 milliseconds) had deteriorated further. For mean RT, players 2 (272.3 milliseconds) and 6 (251.1 milliseconds) had performed better than at baseline, and players 3 (282.9 milliseconds) and 4 (394.1 milliseconds) had improved but had not quite returned to baseline levels.

Discussion

In a series of six concussed Australian Rules footballers, cognitive changes in the 72 hours immediately after a head injury were best characterised as an increase in response variability on a computerised RT test (fig 2). This inconsistency arose from an increase in the proportion of “slow” responses (fig 3), which also resulted in a significant slowing of response latency in these players (fig 1). In contrast, when the cognition of these same six players was assessed with two widely used paper and pencil neuropsychological tests of information processing and psychomotor speed, their performance was observed to improve between the two assessments. Similar improvements were observed in a group of non-injured control players, but these occurred on both paper and pencil and computerised cognitive tests. Post hoc correlational analysis suggested that, in non-injured players, changes in performance between baseline and concussion on computerised and paper and pencil tests were moderately related. However, no such relation was observed in concussed players. These data suggest that computerised tests may be particularly sensitive to the cognitive consequences of sports related concussion, and also that conventional neuropsychological tests do not share this sensitivity in mildly concussed athletes. Computer cognitive tests have many advantages over paper and pencil tests that may allow them to detect subtle impairments such as those expected to occur in mildly concussed athletes.10 These include randomised stimulus presentation, typically high test-retest reliability, lack of floor and ceiling effects, many or possibly even infinite alternative forms, minimisation of tester bias, and the ability to assess a range of cognitive domains in a short period of time. These properties also ensure that a highly accurate estimate of the performance of non-injured players can be gained on repeat testing. In this study we observed no significant change in the response latency of a non-concussed control group between baseline and follow up assessments that were nine weeks apart (fig 1). Further, the distribution of response latencies for this control group was consistent between baseline and follow up assessments (fig 3A). In contrast, the 80th and 90th percentiles of the distribution of response latencies in the concussed players at follow up were significantly slower than expected on the basis of their own baseline performance (fig 3B). This analysis highlights one of the most attractive properties of computerised testing—that is, many responses may be recorded within a very short period of time, and these responses can provide valuable information in specific individuals about the nature of any cognitive impairment after concussion.

Two of the six concussed players were tested while still symptomatic. Interestingly, these players recorded the largest (player 3) and second largest (player 6) increase in response variability from baseline to follow up. Player 3 received the most severe concussion, reporting five symptoms and requiring nine days rest before returning to sport. This player was also the only one to exhibit a decline in performance on the DSST and recorded the greatest increase in response latency on the RT task. This preliminary finding suggests that conventional neuropsychological tests, such as the DSST, may be sufficiently sensitive to detect the cognitive consequences of relatively severe concussions. In cases of mild concussion, more sensitive tests may be necessary to observe cognitive changes. Although these results are promising, this hypothesis requires further validation.

One potential interpretation of the present results is that cognitive deficits after concussion are due to fluctuations in attention and/or information processing, which result in a small (about 10–20%) proportion of abnormally slow psychomotor responses. In turn, this results in increased response variability and a slowing of response latency. This interpretation fits well with the clinical manifestations of concussion, and also with previous studies of response variability in patients with traumatic brain injury.14–16 For example, Stuss and colleagues14, 15 reported that hospital patients with traumatic brain injury display inconsistent responses on simple and choice RT tasks, both within a testing session and between testing sessions. Such attentional fluctuations are unlikely to be detected using paper and pencil neuropsychological tests that give only a single estimate of performance over a very brief period—for example, DSST and TMT. Stuss and colleagues also note that many different and informative analytical techniques may be applied to individual and group performance data from RT tasks.15 In contrast, very few of these techniques may be applied to data from tasks that do not provide estimates of variability.

An alternative explanation is that the concussed players became fatigued towards the end of the 15 minute computerised test (resulting in the observed variability and RT slowing), but that such fatigue effects would not be observed on the shorter (about 90 seconds) paper and pencil tests. We were able to investigate this hypothesis post hoc by analysing latency and variability profiles on each of the three SRT tests for concussed players. Two of the concussed players displayed a pattern of performance that would support this hypothesis, with slowing of RT as the test progressed accompanied by an increase in response variability. However, no evidence of fatigue was observed in the remaining concussed players. Further investigation with larger sample sizes will be necessary to adequately investigate this hypothesis.

As mentioned, it has been proposed previously that computerised cognitive tests will be more sensitive to the effects of concussion in athletes than paper and pencil neuropsychological tests.10, 13 To investigate this hypothesis fully, a much larger sample size than that reported in this study will be required. However, the consistency and magnitude of the impairments observed here in a series of concussed AFL footballers, and the clear dissociation between the serial performance of these athletes and that of non-injured athletes on CogState RT tasks, provides strong preliminary evidence to support this hypothesis. Further support arises from prior findings of increased variability on computerised RT tasks in brain injured patients. As the technical and psychometric sophistication of computerised tasks increases, so too may their sensitivity to the effects of sports related concussion. Important future work will investigate the utility of computerised tasks to aid decisions about recovery and return to play. The data presented here suggest that these tests will be useful for monitoring cognitive function as it returns to baseline after concussion.

Take home message

This paper illustrates the practical problems of using “pen and paper” sideline neuropsychological testing for sport related concussion. Computerised test batteries, such as CogState, both allow a wider range of domains to be tested and have the ability to detect fatiguability or variability in cognitive performance, which in turn is a key measure of recovery.

References

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.