Intended for healthcare professionals

Endgames Statistical Question

Explanatory trials versus pragmatic trials

BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g6694 (Published 13 November 2014) Cite this as: BMJ 2014;349:g6694
  1. Philip Sedgwick, reader in medical statistics and medical education1
  1. 1Institute for Medical and Biomedical Education, St George’s, University of London, London, UK
  1. p.sedgwick{at}sgul.ac.uk

Researchers assessed the effectiveness of weekly delivery of low dose high frequency therapeutic ultrasound in conjunction with standard care for hard to heal venous leg ulcers. A multicentre pragmatic two arm randomised controlled trial study design was used. Ultrasound was delivered during weekly dressing changes, and the duration depended on the size of the ulcer. The control treatment was standard care alone, comprising low adherent dressings and four layer bandaging that was high compression, reduced compression, or no compression, depending on the patient’s tolerance. Treatment was delivered for a maximum of 12 weeks.1

Participants were 337 patients with at least one venous leg ulcer of more than six months’ duration or greater than 5 cm2 in area, and an ankle brachial pressure index of 0.8 or more. The primary outcome was time to healing of the largest eligible leg ulcer. Secondary outcomes included health related quality of life and adverse events. Participants were followed for a maximum of 12 months.

The researchers reported that low dose high frequency ultrasound given weekly for 12 weeks during dressing changes in addition to standard care did not significantly increase ulcer healing rates, affect quality of life, or reduce ulcer recurrence.

Which of the following statements, if any, are true?

  • a) As a pragmatic trial, it would have been undertaken in a routine clinical and healthcare setting

  • b) The participants would have been monitored closely to ensure adherence to their allocated treatment regimen

  • c) The trial would be expected to have high external validity

  • d) The trial would be expected to have high internal validity

Answers

Statements a and c are true, whereas b and d are false.

Clinical trials are described as explanatory or pragmatic. Explanatory trials generally measure efficacy—that is, the benefit of a treatment under ideal conditions. The principal aim is to establish whether a treatment works. Explanatory trials typically take place during the initial development of an intervention—for example, a phase II trial that may be placebo controlled. Participants are carefully selected, and the sample will be a homogeneous group with respect to their characteristics. Participants typically have well defined characteristics for the clinical condition of interest, and those with associated comorbidities will tend to be excluded to minimise confounding when evaluating treatment effects. Furthermore, participants may all be of the same sex, or within a relatively small age range, to minimise confounding further. Follow-up with research staff is intensive and participants will generally attend mandatory follow-up visits in a research clinic. Treatment is strictly enforced and participants are monitored closely to ensure adherence to their allocated treatment regimen.

Although explanatory trials are essential for ascertaining whether a new intervention works, they have limited generalisability to clinical practice, where patients are diverse in their characteristics and may not adhere to prescribed treatment. So called pragmatic trials are undertaken to provide estimates of treatment effects that are generalisable to clinical practice. A pragmatic randomised controlled trial study design was used in the above study. The aim of the trial was to assess the effectiveness of weekly delivery of low dose high frequency therapeutic ultrasound in conjunction with standard care for hard to heal venous leg ulcers.

Pragmatic trials measure effectiveness—that is, the benefit of treatment in clinical practice. Typically the aim of a pragmatic trial, as in the example above, is to help clinicians decide between a new intervention and current best treatment. Pragmatic trials take place in a routine healthcare or clinical practice setting (a is true), and participants may be receiving other healthcare. The setting for the above trial was community and district nurse led services, community leg ulcer clinics, and hospital outpatient leg ulcer clinics in 12 urban and rural settings. More generally, pragmatic trials are designed so that they incorporate the variations seen between patients. The participants reflect those seen in clinical practice to whom the treatments will be applied. Typically there are few, if any, selection criteria other than the presence of disease or diagnosis of interest. The participants will be a heterogeneous group with respect to their characteristics. Once allocated to a treatment group, patients do not necessarily all receive the same treatment regimen. The treatment regimen may be tailored to the patient’s needs. In the above study, for those patients allocated to the intervention, the duration of ultrasound depended on the size of the ulcer. Although all patients received standard care, the amount of compression applied with four layer bandaging depended on the patient’s tolerance. Moreover, follow-up in a pragmatic trial will generally not be as intense as in an explanatory trial. Participants are not followed closely to ensure treatment adherence—flexibility is applied as in clinical practice (b is false). Furthermore, participants may withdraw consent at any time after randomisation; they may request a different treatment and possibly the alternative one in the trial to which they were allocated.

Both explanatory and pragmatic trials incorporate random allocation of participants to the treatment group, thereby promoting internal validity. Described in a previous question,2 internal validity is the extent to which the observed treatment effects can be ascribed to differences in treatment and not confounding, thereby allowing the inference of causality to be ascribed to the treatment. Explanatory trial designs have high internal validity because the participants are a highly selected group of patients and are monitored closely to ensure adherence to their allocated treatment regimen. However, because the participants are a highly selected group of patients, explanatory trials tend to have limited external validity.2 External validity is the extent to which the study results can be applied to patients other than those studied and to whom the treatments will be applied in clinical practice. By contrast, the pragmatic trial in the above example would be expected to have high external validity (c is true) because the trial participants were recruited from patients seen in routine clinical practice. However, the trial would not be expected to have high internal validity (d is false). In particular, the internal validity for a pragmatic design will be lower than that for an explanatory design; the participants in a pragmatic trial are a heterogeneous group and may not adhere to the treatment regimen they are allocated. For that reason, the analysis of pragmatic trials will be based on an intention to treat approach so that confounding is minimised when evaluating treatment effects.3

In the above trial, because the participants would have been aware of their treatment allocation, there was the potential for response bias in the outcome measures.4 It was reported that the investigators who assessed the outcome measures were blind to treatment allocation, thereby minimising assessment bias. A sham treatment could have been used to minimise such biases.5 However, pragmatic trials tend not to incorporate sham treatments or placebos. Typically, the aim of a pragmatic trial is to help clinicians decide between a new intervention and current best treatment. It may be argued that a sham treatment or placebo is not pragmatic, not least because it will not be used in clinical practice. Nonetheless, for patients to have knowledge of their treatment allocation in a pragmatic trial might be part of their treatment, as it is in clinical practice.

In pragmatic trials, the outcome measures are typically clinical assessments. They tend to be patient centred and to measure a patient’s symptoms, overall mental state, and quality of life. Emphasis is placed on measuring the effects of a disease or condition on how a patient functions. In explanatory trials the outcomes tend to be surrogate markers. The purpose of a surrogate marker is that it should be closely related to a clinical outcome, and as the name suggests it is a proxy measure. Surrogate markers tend to be quicker and cheaper to measure than clinical outcomes. Surrogate endpoints can include the measurement of a biomarker—a characteristic that is measured objectively and that is an indicator for understanding the biological basis of the response to treatment. For example, in an explanatory trial investigating the efficacy of treatment for the healing of leg ulcers, a suitable biomarker might be bacterial load or the presence of a specific bacterium. As such, surrogate markers have less relevance to patients and healthcare practitioners than clinical outcomes.

Although trials are labelled as explanatory or pragmatic, it is generally recognised that there is a continuum rather than a dichotomy in definition and study design. The usefulness of a trial’s results lies in the generalisability to patients other than the participants studied in the trial. Tools have been created to help clinicians, study reviewers, and policy makers to ascertain the external validity of a trial and the degree of treatment effectiveness that can be expected in different healthcare and clinical settings.

Notes

Cite this as: BMJ 2014;349:g6694

Footnotes

  • Competing interests: None declared.

References

View Abstract