Reporting attrition in randomised controlled trials

Effects of attrition

Attrition can introduce bias if the characteristics of people lost to follow-up differ between the randomised groups. In terms of bias, this loss is important only if the differing characteristic is correlated with the trial's outcome measures. However, attrition is not a black and white issue—there is no specific level of loss to follow-up at which attrition related bias becomes acknowledged as a problem. Schulz and Grimes argue that loss to follow-up of 5% or lower is usually of little concern, whereas a loss of 20% or greater means that readers should be concerned about the possibility of bias; losses between 5% and 20% may still be a source of bias.3 For the purposes of this article we will not differentiate between loss to follow-up and missing data. We have also not considered exclusions by trial investigators. Although exclusion is justified in some cases,3 generally it is ill advised.1 2

Reporting of attrition

In a review of trials published in four general medical journals in 2002, 54% (71) of the 132 trials had some loss to follow-up for the main analysis.4 Among these trials the median percentage loss was 7% (minimum 0.08%, maximum 48%; interquartile range 2-18%). These data suggest that potentially problematic loss to follow-up occurs in many trials, even those published in high quality general journals.4

The standard practice for reporting trials, as encouraged by CONSORT,5 is to include a table describing the baseline characteristics of the trial participants. This table provides useful information on all participants and confirms the success of the randomisation process. However, if there has been loss to follow-up, information from the whole sample may not adequately describe the analysed sample nor accurately reflect the comparability of the trial groups. Thus, we suggest it is informative to present baseline characteristics for the participants for whom data have been analysed and those who are lost to follow-up separately. This would provide a clearer picture of the subsample not included in an analysis and may help indicate potential attrition bias. As an example we have taken data from a recently published randomised trial.

Is your trial fall proof?

Credit: http://www.safehip.com/

Example trial

A trial of hip protectors for preventing hip fracture is typical of many studies in that about 20% of participants were lost to follow-up from routine data collection.6 The authors dealt with this problem for the main outcome (hip fracture) by accessing the general practice records of non-responders. However, for secondary outcomes (such as quality of life) this was not possible. Therefore, reports of these secondary outcomes are at a high risk of bias from attrition. We can assess whether these outcomes may be affected by attrition bias by comparing rates of loss to follow-up between the arms of the trial as well as by examining the baseline characteristics of participants who were lost to follow-up and the characteristics of those remaining.

The trial had a small but significant difference in attrition rates between the two arms (372 (28%) in the intervention group and 619 (22%) in the control group, P = 0.001). This is the first indication of a potential problem (table). Because we have different numbers of participants leaving the trial arms, the likelihood that participants in one group are not balanced with similar participants in the other trial arm is increased.

The table includes the baseline characteristics of all the participants. Not surprisingly, the baseline characteristics of the whole sample are well balanced, as we would expect through random allocation. However, these data cannot tell us whether the sample of women included in the analysis is balanced between the two treatment groups.

Of more interest is a comparison of the baseline characteristics of those who have left the study and those remaining. The table shows, as you might expect, that the between group differences in those lost to follow-up tend to be larger than any chance differences at baseline. For example, more volunteers, people with poor or fair health, and people with a previous fracture are lost from the control group than the intervention group. In these examples, the differential attrition leads to varying changes in the characteristics of those participants remaining in the trial when compared with all participants at baseline (table). This information is useful for the reader. Yet, providing information about the participants who have not contributed to the analysis may also be informative.

Interpreting results

The internal validity of a trial's results partly depends on the between group balance in prognostic characteristics of those who remain in the trial. In addition, important imbalances that are not readily apparent in the analysed groups may become apparent when we examine the between group characteristics of those lost to follow-up.

To clarify this point we have carried out some simulation work to assess the effect of attrition bias on baseline characteristics and the type 1 error rate. The simulations used a population of trials each with 630 participants and 10% biased attrition in one arm and 10% random attrition in the other. Although the type 1 error rate was substantially increased, such attrition does not always lead to an apparent imbalance of the baseline characteristics of the participants remaining in the trial. Thus, assessment of the characteristics of those lost to follow-up may be particularly important. Further work is needed to examine how this issue is influenced by sample size and other factors.

The data in the table would allow trialists to make a qualitative judgment about whether an important predictor variable has become more imbalanced since randomisation. They could then decide whether to carry out a sensitivity analysis treating a variable, such as previous fracture, as a covariate. However, since such judgment is subjective, especially for smaller sample sizes, a statistical test could be useful.

Statistical testing

The use of statistical tests in this context is complex, and there are arguments for and against testing. From current knowledge we suggest it is not useful to test for differences statistically. A further reason for avoiding statistical testing stems from arguments Altman and others have put forward against baseline testing of the total randomised group—that imbalance of a predictor variable may still bias the study results, even if the imbalance does not reach conventional levels of significance.7^–10

Decisions about covariates are normally made before the start of the trial. However, because attrition bias cannot always be anticipated, information on differential attrition is relevant at the analysis stage. Nevertheless, adjusting for variables that have not been specified in advance is poor statistical practice and may introduce bias.11 As Altman suggests, if serious attrition bias is suspected, the analysis should be carried out as originally planned, with perhaps a second analysis adjusting for the new covariate.11 Furthermore, the data presented can only include observed baseline variables: unrecorded or unknown variables may also be imbalanced.

Implications

Baseline tables are a useful way of assessing the study sample, but the characteristics of the sample may change during the study, especially by attrition. As the hip protector example shows, the baseline characteristics of those lost to follow-up during a study can differ. And since attrition of data commonly occurs in trials,1 2 4 such differences could also be common. We know that missing data make it more difficult to carry out a true intention to treat analysis. Yet, since information on the baseline characteristics of those lost to follow-up and those for whom data are analysed is rarely reported, it is almost impossible to identify the effect of attrition on the study sample as a whole and therefore the result of the randomised controlled trial.

Questions arise about when these modified baseline tables could and should be used. We suggest that information on the participants included in the main analysis of a paper is of interest, especially if attrition is high. Missing data are more common for secondary outcome measures, as researchers often focus on collecting data on the primary outcome. Authors could consider presenting a table showing baseline characteristics of those who were and were not analysed when reporting secondary outcomes. The table might look different for different analyses of the same study. Although this table would require increased journal space, the information is arguably more useful than that in a standard baseline table. Since many journals publish online, these tables could be made available to interested readers in the electronic version only. Alternatively, the information might be incorporated into the flow diagram recommended by CONSORT.5