• Reliable Change index
• concussion
• neuropsychology
• head injury

Reliable Change (RC) indices are a group of statistical techniques used in many areas of medicine to help to determine when an individual’s performance on a neuropsychological test has changed from a previous assessment¹ with the same test. Recently, in sports concussion, numerous authors have advocated the application of RC analyses to neuropsychological test data collected at baseline (preseason) and after a concussion.^2,3 These authors have stated that the results of RC analyses provide the best means for guiding decisions about whether or not true change in cognitive function has occurred after a concussion, and can therefore assist the return to play decision making process. Although we support the use of RC techniques to guide decisions about concussion, we have concerns about the statistical computation and interpretation of various RC indices.

RC techniques were first described by Jacobson and Traux,¹ and were designed to aid decision making about the significance of cognitive changes in patients in whom an injury or intervention had taken place. These and subsequent authors^4,5 proposed that the most efficient way of determining whether an individual’s score on a specific cognitive measure had changed was to express the magnitude of change—that is, a change score—as a function of the normal variation found for that measure. Normal variation in performance on the cognitive measure was estimated from a group of similar subjects in whom no injury or intervention had occurred. Mathematically, the individual’s change in performance is expressed in the numerator, and the normal variation in performance on that measure is expressed in the denominator as follows.

Step 1: Calculate the standard error of measurement (S_E)

Step 2: Calculate the standard error of difference (SE_diff)

Step 3: Calculate the RC score

x₁ is the participant’s baseline score, x₂ is the participant’s follow up score, SE_diff is the standard error of the difference, S_E is the standard error of measurement, S₁ is the standard deviation of the control group at baseline, and r₁₂ is the test-retest reliability.

Clinicians, neuropsychologists, and statisticians working with RC techniques soon realised that “true” changes in test scores could be obscured by performance changes due to practice—that is, prior exposure to a test leads to improved performance on a subsequent assessment—and also by statistical phenomena such as the reliability of the test⁶ itself and the related regression to the mean. This has led to the description⁷ and application^2,3 of several variants of the basic RC index. These variants have sought to provide more accurate guidance to decisions about change caused by an event by incorporating corrections for practice effects, test reliability, and regression to the mean.

The outcome of RC analyses may be interpreted statistically as a z score, with changes greater than 1.96 indicating that true change has occurred. In sport medicine, where the focus is to detect decline in performance after a concussion—that is, a one tailed hypothesis—an RC of less then −1.65 indicates that true decline has occurred.³ One advantage of RC statistics is therefore that they can be applied immediately to individual level data, and therefore interpreted on an individual basis. This makes them applicable to clinical situations such as sports related concussion.

RC analyses were designed in accordance with conventional models of neuropsychological assessment—that is, to determine whether the change observed in the individual is true by comparing it with change that occurs normally in some matched normative data set. The problem with currently applied RC calculations is that the normal amount of variability in change over time within individuals is estimated on the basis of differences between individuals assessed at a single time point! There is no reason to believe that variation between individuals at one time point accurately represents the variation within individuals between two time points. A related problem with current RC analyses is that the normal variation represented in the denominator is termed the standard error of the difference (SE_diff),¹ despite the fact that it is computationally the standard deviation of the individual scores at one point in time. A true estimate of change requires the standard deviation of difference scores (SD_diff) in the denominator.

In sports medicine, we are in the fortunate position of having many healthy young subjects enrolled in longitudinal studies of concussion, and relatively few neuropsychological measures administered in these studies. There should be no reason why the normal change in performance over time within individuals cannot be determined directly from such control group data rather than using inappropriate estimates of variation. In fact, many researchers have obtained serial data for inclusion in RC calculations as corrections for the effects of practice observed in normal populations, including some working in sports concussion.^2,3 Although such serially acquired data are adequate for directly estimating the SD_diff from a normal sample, these authors have continued to use the “estimated” SD_diff rather than directly calculating the SD_diff for inclusion in the RC calculation.

Some minor alterations to previous RC calculations produces an RC calculation that is mathematically and theoretically correct, yet retains all the virtues of previously proposed RC calculation. The alterations are as follows.

Step 1: Calculate the difference scores for each individual in a control group assessed at an appropriate test-retest interval.

Step 2: Calculate the sum of the squared (SUM_SQ) deviations from the mean difference score. This will be included in the calculation for the standard deviation of the difference score.

Step 3: Calculate the standard deviation of these difference scores (SD_diff). This becomes the denominator in the RC equation.

Step 4: Calculate the RC score for the individual athletes by placing the individual athlete’s change score in the numerator of the RC equation, and the SD_diff score in the denominator.

A_diff is the test-retest difference score for person A, B_diff is the test-retest difference score for person B, N_diff is the test-retest difference score for person N, μ_diff is the mean difference score for the control group of size N, N is the total number of paired observations, x₁ is the concussed athlete’s baseline test score, and x₂ is the concussed athlete’s test score after concussion.

This RC technique can be interpreted as a z score, with a change of greater than −1.65, indicating significant decline from baseline using a one tailed hypothesis. Such RC scores may also be interpreted as “effect size” calculations, very similar to Cohen’s d scores as described by Zakzanis.⁸ Our research group applies this calculation to neuropsychological test data gained in concussed athletes in many sports world wide and in many other medical applications where issues of change in an individual’s cognitive status are pertinent.^7,9 Corrections for practice effects and other confounding variables may also be included in this calculation as per current RC techniques.