You are here

Article Text

Article menu
XML
Public health
Research
Reporting of harms data in RCTs: a systematic review of empirical assessments against the CONSORT harms extension
  1. Alex Hodkinson,
  2. Jamie J Kirkham,
  3. Catrin Tudur-Smith,
  4. Carrol Gamble
  1. Department of Biostatistics, MRC North West Hub for Trials Methodology Research, University of Liverpool, Liverpool, UK
  1. Correspondence to Alex Hodkinson; ahoddy{at}liverpool.ac.uk

Abstract

Objective To determine the standard of reporting of harms-related data, in randomised controlled trials (RCTs) according to the Consolidated Standards of Reporting Trials (CONSORT) statement extension for harms.

Design Systematic review.

Data sources The Cochrane library, Ovid MEDLINE, Scopus and ISI Web of Knowledge were searched for relevant literature.

Eligibility criteria for selecting studies We included publications of studies that used the CONSORT harms extension to assess the reporting of harms in RCTs.

Results We identified 7 studies which included between 10 and 205 RCTs. The clinical areas of the 7 studies were: hypertension (1), urology (1), epilepsy (1), complimentary medicine (2) and two not restricted to a clinical topic. Quality of the 7 studies was assessed by a risk of bias tool and was found to be variable. Adherence to the CONSORT harms criteria reported in the 7 studies was inadequate and variable across the items in the checklist. Adverse events are poorly defined, with 6 studies failing to exceed 50% adherence to the items in the checklist.

Conclusions Readers of RCT publications need to be able to balance the trade-offs between benefits and harms of interventions. This systematic review suggests that this is compromised due to poor reporting of harms which is evident across a range of clinical areas. Improvements in quality could be achieved by wider adoption of the CONSORT harms criteria by journals reporting RCTs.

  • General Medicine (see Internal Medicine)
  • Clinical Trial Methodology
  • Biostatistics

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/3.0/

https://doi.org/10.1136/bmjopen-2013-003436

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Article summary

    Strengths and limitations of this study

    • This is the first study to systematically review empirical studies assessing the quality of reporting according to the CONSORT-harms guideline.

    • The review was strengthened by its assessment of quality of the included studies across four key domains.

    • This study should be regarded as a reflection of reporting standards in general rather than an assessment of adherence to the CONSORT-harms extension.

    • Some included studies contained trials reported prior to the publication of the CONSORT harms guideline; we did not extract these results.

    • We have not assessed changes in reporting over time.

    Introduction

    Every healthcare intervention is associated with a risk of harmful or adverse events, that must be balanced against the potential favourable outcomes.1

    The Consolidated Standards of Reporting Trials (CONSORT) statement aims to improve the quality of published reports of randomised controlled trials (RCTs) and has been widely endorsed by healthcare journals, leading to improvements in quality when used by manuscript authors and peer reviewers.2–4 However, some reports suggest that assessment and reporting of harms in clinical trials may be suboptimal.5–7

    The standard CONSORT statement8 is primarily aimed at reporting the intended, usually beneficial effects of intervention(s) with only one item (item 19) devoted to unintended adverse events (harms) in the original 2001 checklist. Owing to accumulating evidence that reporting on harms-related data in RCTs was of poor quality with an imbalanced ratio of benefit–harms reporting, a CONSORT statement extension was developed in 2004 to improve harms reporting (CONSORT-harms) and to help address perceived shortcomings in measurement, analysis and reporting of harms data.9 The subsequent update of the standard CONSORT statement, published in 2010,10 now specifically refers to the additional CONSORT-harms extension but it is still unclear whether authors and journals routinely adopt the use of this extension. The aim of this paper is to systematically review the evidence from previously conducted empirical studies that have assessed the adequacy of harms reporting in RCTs using the CONSORT-harms extension as a benchmark.

    Methods

    A protocol for the systematic review was developed by AH, CTS, CG and JJK.

    Study inclusion criteria

    We included published and unpublished research that evaluated the quality of harms reporting in RCTs against the CONSORT-harms recommendations.9 No restriction was placed on the clinical area or type of intervention studied. Excluded studies were those that assessed harms reporting using assessment criteria other than CONSORT-harms and studies that assessed harms reporting using study designs for which the CONSORT guideline was not intended (eg, observational studies).

    Identification of studies

    AH, CTS and CG developed the search strategy with support from an information specialist which was then implemented by AH in the following databases: the Cochrane Methodology register, Database of Abstracts of Reviews of Effects (DARE), Ovid MEDLINE, Scopus and ISI Web of Knowledge. Conference abstracts were searched for in the Web of Knowledge Conference Proceedings Citation Indexes (CPCI-S or CPCI-SSH) and the Zetoc database.11 An unpublished Masters dissertation involving one of the authors (JJK) was also obtained. Date filters were not used during the search criteria; our interest lies only within reviews published after 2004, with the cut-off date June 2012.

    Titles and abstracts of reports identified by the search were screened by AH and full articles obtained for all potentially eligible studies. Each full article was assessed independently by two reviewers (AH and CTS) to determine eligibility.

    Quality assessment

    Two reviewers (AH and JJK) independently assessed the methodological quality of each study using the Cochrane Risk of Bias (RoB) tool12 as a guideline to cover the following aspects. Criteria were graded as low risk, high risk or unclear as indicated.

    1. Were the trials included in the study a representative sample, for example, unselected journals, and reasonable time scale?

      Low risk of bias: studies included trials from a primary search of all the available literature.

      High risk of bias: studies were highly selective of the trials included, for example, high-impact journals or specialised-journals only.

      Unclear risk of bias: not stated how studies were selected.

    2. During the data extraction of CONSORT-harms criteria, were reviewers blinded to study authors, institution, journal name and sponsors?

      Low risk of bias: reviewers were blinded.

      High risk of bias: reviewers were not blinded.

      Unclear risk of bias: not stated.

    3. Is there evidence of selective outcome reporting in the study (ie, were all CONSORT-harms recommendations considered and if not were suitable reasons provided)?

      Low risk of bias: studies that considered all CONSORT-harms criteria or reasons for excluding specific criteria were transparent and justified.

      High risk of bias: studies did not consider all CONSORT-harms criteria.

      Unclear risk of bias: unclear whether all CONSORT-harms criteria were considered.

    4. Did more than one reviewer assess the CONSORT-harms criteria for each primary RCT, with a description of how agreement was achieved?

      Low risk of bias: data extraction was completed independently by two people or reasonable attempts were made to maximise data extraction reliability.

      High risk of bias: data extraction not completed independently by two people.

      Unclear risk of bias: not stated.

    Data collection and extraction

    Two reviewers (AH and JJK) independently extracted the data and any discrepancies were resolved through a consensus discussion with a third reviewer (CTS). Data extraction included

    • Study characteristics: inclusion criteria including clinical area, types of interventions, databases or journals searched within the study and any search date restrictions.

    • Sample size (defined by the number of RCT reports assessed for reporting quality).

    • Reporting quality: inclusion of any of the 10 recommendations from the 2004 CONSORT-harms checklist (table 1 and supplemental data: see CONSORT plots).

    View this table:
    Table 1

    The 10 CONSORT-harms recommendations9

    Lead authors were contacted through email with any queries relating to the quality assessment or data extraction.

    Data analysis and presentation

    For each study, the percentage of included RCTs that satisfied each CONSORT-harms recommendation is presented with 95% CIs. Some studies had presented data for individual items described within each of the 10 criteria rather than overall data. These are presented as such in tables with footnotes to provide further explanation. Forest plots were used to graphically depict the levels of adherence to the CONSORT harms recommendations so that readers can easily discern the extent of compliance and heterogeneity between studies with the I2 statistic (included as supplementary material online). We refrained from statistically combining results from the different studies due to the differences in their study characteristics. In accordance with the Cochrane Handbook, I2 statistics were interpreted as (0–40%, might not be important; 30–60%, may represent moderate heterogeneity; 50–90% may represent substantial heterogeneity; 75–100%, considerable heterogeneity).13

    Results

    The search strategy identified 5083 potentially eligible study cohorts from which seven studies assessing the quality of reporting across almost 800 RCTs were included (figure 1).

    Figure 1
    Figure 1

    Flow diagram of study identification and selection.

    Five studies14–18 (with the study19 recently published) contained trials focusing on specific clinical areas with two20 ,21 covering multiple clinical areas (table 2). Four studies14 ,17 ,20 ,21 included trials using drug interventions, one comparing acupuncture18 and another alternative complementary medicines,16 the interventions were unclear in one study.15 MEDLINE was used by four of the studies17 ,18 ,20 ,21 to identify the relevant literature; three14 ,16 ,17 used the Cochrane database of RCTs and three15–17 searched specialised-journal databases. The date restrictions used in the search strategy of each study ranged from a 1 year period up to a 9 years span. The studies were published after 2008, 4 years after the release of the harms extension with three15 ,17 ,21 of them including trials that had been published before the publication of CONSORT-harms. Five studies14–17 ,20 excluded trials published in a non-English language.

    View this table:
    Table 2

    Characteristics of included reviews

    Risk of bias

    Lead authors were contacted by email with any queries relating to the quality of their study, or CONSORT criteria; however, two authors failed to respond.15 ,18 The risk of bias for the seven included studies, assessed across four domains, is summarised in table 3. Six studies14 ,15 ,17 ,18 ,20 ,21 were classified as high risk for bias for at least one domain with two of these studies14 ,20 classified as high risk for three domains. Four studies13 ,14 ,20 ,21 did not include trials of a representative sample targeting specific journals rather than a database search. Blinding of assessors was only implemented in two studies15 ,16 with one unclear.18 Most studies used all the CONSORT-harms criteria with the exception of the subgroup analysis item; one study16 however, discarded the use of recommendation eight, since it was captured elsewhere within the data extraction, and recommendation 10, which was considered too vague to assess with any objectivity. Reporting of the assessment within three15 ,18 ,20 of the seven identified studies was unclear and authors were contacted. The authors did not respond for two studies15 ,17 and in another study20 a response was received but some details remained unclear. Six studies15–18 ,20 ,21 had used two independent data extractors while one study14 had not and was classified as high risk of bias for this domain.

    View this table:
    Table 3

    Risk of bias assessment

    CONSORT harms recommendations

    Results extracted for the CONSORT-harms criteria (table 4) demonstrate variability in the level of adherence to items. Heterogeneity is highlighted by the individual Forest plots where inflated I2 values of over 85% are represented for all recommendations, denoting considerable heterogeneity.

    View this table:
    Table 4

    CONSORT harms criteria reported across included reviews

    Of the six studies that assess inclusion of harms in the title and abstract of their included RCTs, three16 ,20 ,21 reported compliance in over 70% of RCTs, but three14–16 reported compliance in less than 30% of RCTs. The introduction section of the included RCTs reflect an imbalance in the reporting benefit–harms, with one study16 reporting that less than 5% of RCTs had mentioned harms in the introduction, and one study17 reporting more than 70% of its included RCTs has satisfied this criteria.

    The definition of adverse events in reports is unsatisfactory with most studies14–16 ,17 ,20 indicating that fewer than 20% of RCTs satisfy these criteria adequately. The collection of harms-related information is described by more than 80% of RCTs in two studies,20 ,21 but this high level is not consistent across the other five studies with one study14 suggesting that as few as 10% of RCTs had provided an adequate description. The analysis and coding of adverse events is poorly described, with less than 50% of RCTs satisfying this criteria across six studies,13 ,16–18 ,20 ,21 with one of these studies13 indicating that none of the RCTs had provided an adequate description. The reporting of participant withdrawals due to harms was inconsistent with two studies15 ,16 suggesting infrequent reporting (less than 40% of RCTs had mentioned withdrawals), three studies13 ,20 ,21 suggesting occasional reporting (50–60% of RCTs had mentioned withdrawals) and two studies17 suggesting that reporting of withdrawals was quite common (approximately 70% of RCTs had mentioned withdrawals).

    When providing the denominators within trial reports, the results were also varied across studies, with three17 ,20 ,21 studies identifying more than 70% of trials that satisfied this criterion, but two studies13 ,15 identifying less than 20% adherence. The risk and severity grading of adverse events, is detailed in more than 70% of trial across two studies,20 ,21 but the reporting is inadequate in three studies.13 ,15 ,17 An assessment of reporting of harms within subgroup analysis was only carried out within one study.21

    Four studies14 ,15 ,17 ,21 assessed their included RCTs for a balanced report on the benefits and harms within their discussion: one study13 identified a very low percentage (<10%), two studies14 ,16 identified a moderate percentage (approximately 60%), and one study21 identified a high percentage (over 80%) of trials that met this criterion.

    Discussion

    Summary of findings

    This is the first study to systematically review empirical studies assessing the quality of reporting according to the CONSORT-harms guideline.9 Data were extracted from seven studies that had each assessed the quality of reporting across almost 800 RCTs from a range of clinical specialities. Eight years have now passed since the release of the harms extension, allowing adequate time for the guideline implementation. This review highlights that the reporting of harms in RCTs is inconsistent, and at times very poor. Heterogeneity is easily discerned between studies for each recommendation. Further adherence to the CONSORT-harms is needed.

    The standard CONSORT is well established in health research with building evidence to support the use of the guideline.5 ,6 Currently the standard CONSORT is endorsed by over 50% of the core medical journals in the abridged Index Medicus on PubMed.22 In a review23 of 116 health research journals, 41 provided online instructions to authors. Almost half (19/41 (46%)) mentioned the standard CONSORT guideline but none referred to the CONSORT extension for harms.

    Strengths and weaknesses of the study

    In this study we have focused on assessing reporting according to the CONSORT-harms criteria only. The included studies contained trials reported prior to the publication of the CONSORT-harms guideline. However, we have not assessed changes in reporting over time. Nevertheless, our results support those from previous studies3 ,4 that used various guidelines published before the release of the CONSORT-harms extension. This study should be regarded as a reflection of reporting standards in general rather than an assessment of adherence to the CONSORT-harms extension.

    This review was strengthened by its assessment of quality of the included studies across four key domains. With the guidance of the Cochrane review12 we have designed a RoB tool to perform a generalisable assessment of the included studies. In this assessment only the one study15 demonstrated low RoB across all four of the assessment criteria. No restriction was placed on the inclusion criteria of the identified studies such that the time span and clinical areas of their included studies varied. While this is a-strength in terms of generalisability of results, it may also be considered as a level of heterogeneity that cannot be explored due to the limited number of studies.

    Conclusions and implications

    Complete and accurate reporting is essential to guide decisions on advances in medical interventions. The responsibility to ensure greater balance between reporting of both benefits and harms lies with authors of research and journals publishing that research. We recognised that journals have limited space for the reporting of all outcomes which can lead to selective outcomes reporting.24 ,25 We recommend the use of supplementary online tables to help summarise key results on harms.

    Further dissemination strategies should be used to ensure that trial journal editors and trial investigators are aware of the importance of adequate reporting of harms-related data in RCTs. As it stands, it is unclear as to whether the problem of the poor reporting of harms data in trial publications is a result of the lack of awareness of the CONSORT for harms statement, or journals and peer reviewers not implementing this guideline. The most effective strategy would follow that of the CONSORT statement with the extension for harms comprehensively incorporated in journal requirements along with clear instructions to peer reviewers for guidelines of acceptance.

    Acknowledgments

    The authors are grateful to Su Golder and Fiona Beyer York University for expanding on the item words used within the databases and Alison Beamond University of Liverpool, for recommending databases to search conference abstracts.

    References

    1. ↵
      1. Cuervo LG,
      2. Aronson JK
      . The road to health care. BMJ 2004;329:1–2.
      OpenUrlFREE Full Text
    2. ↵
      1. Derry S,
      2. Loke YK,
      3. Aronson JK
      . Incomplete evidence: the inadequacy of databases in tracing published adverse drug reactions in clinical trials. BMC Med Res Methodol 2001;1:7.
      OpenUrlCrossRefPubMed
    3. ↵
      1. Loke YK,
      2. Derry S
      . Reporting of adverse drug reactions in randomised controlled trials—a systematic survey. BMC Clin Pharmacol 2001;1:3.
      OpenUrlCrossRefPubMed
    4. ↵
      1. Ioannidis JP,
      2. Lau J
      . Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA 2001;285:437–43.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Moher D,
      2. Jones A,
      3. Lepage L,
      4. et al
      . Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA 2001;285:1992–5.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Plint AC,
      2. Moher D,
      3. Morrison A,
      4. et al
      . Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 2006;185:263–7.
      OpenUrlPubMedWeb of Science
    7. ↵
      1. Cobo E,
      2. Cortes J,
      3. Ribera JM,
      4. et al
      . Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. BMJ 2011;343:d6783.
      OpenUrlAbstract/FREE Full Text
    8. ↵
    9. ↵
      1. Ioannidis JP,
      2. Evans SJ,
      3. Gotzsche PC,
      4. et al
      . Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141:781–8.
      OpenUrlPubMedWeb of Science
    10. ↵
      CONSORT statement. 2010. http://www.consort-statement.org/consort-statement/ (accessed Dec 2011).
    11. ↵
      Zetoc: Informing Research. http://www.zetoc.mimas.ac.uk/
    12. ↵
      Turner L, Shamseer L, Altman DG, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 2012;11:MR000030.
    13. ↵
      1. Higgins J,
      2. Green S
      . eds Cochrane handbook for systematic reviews of interventions. Version 5.1.0. The Cochrane Collaboration. 2011. http://handbook.cochrane.org/ (accessed 3 Jan 2013).
    14. ↵
      1. Bagul N,
      2. Kirkham JJ
      . The reporting of harms in randomised controlled trials of hypertension. 2011.
    15. ↵
      1. Breau RH,
      2. Gaboury I,
      3. Scales CD Jr.,
      4. et al
      . Reporting of harm in randomized controlled trials published in the urological literature. J Urol 2011;183:1693–7.
      OpenUrl
    16. ↵
      1. Turner LA,
      2. Singh K,
      3. Garritty C,
      4. et al
      . An evaluation of the completeness of safety reporting in reports of complementary and alternative medicine trials. BMC Complement Altern Med 2011;11:67.
      OpenUrlCrossRefPubMed
    17. ↵
      1. Shukralla AA,
      2. Tudur-Smith C,
      3. Powell GA,
      4. et al
      . Reporting of adverse events in randomised controlled trials of antiepileptic drugs using the CONSORT criteria for reporting harms. Epilepsy Res 2011;97:20–9.
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Capili B,
      2. Anastasi JK,
      3. Geiger JN
      . Adverse event reporting in acupuncture clinical trials focusing on pain. Clin J Pain 2010;26:43–8.
      OpenUrlPubMed
    19. ↵
      1. Bagul NB,
      2. Kirkham JJ
      . The reporting of harms in randomized controlled trials of hypertension using the CONSORT criteria for harm reporting. Clin Exp Hypertens 2012;34:548–54.
      OpenUrlCrossRefPubMed
    20. ↵
      1. Pitrou I,
      2. Boutron I,
      3. Ahmad N,
      4. et al
      . Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med 2011;169:1756–61.
      OpenUrl
    21. ↵
      1. Haidich AB,
      2. Birtsou C,
      3. Dardavessis T,
      4. et al
      . The quality of safety reporting in trials is still suboptimal: survey of major general medical journals. J Clin Epidemiol 2011;64:124–35.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      CONSORT Statement. CONSORT endorsers—journals. http://www.consort-statement.org/about-consort/consort-endorsement/consort-endorsers–-journals/ (accessed 4 Aug 2011).
    23. ↵
      1. Hirst A,
      2. Altman DG
      . Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS ONE 2012;7:e35621.
      OpenUrlPubMed
    24. ↵
      1. Kirkham JJ,
      2. Dwan KM,
      3. Altman DG,
      4. et al
      . The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Smyth RMD,
      2. Kirkham JJ,
      3. Jacoby A,
      4. et al
      . Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists. BMJ 2011;342:c7153.
      OpenUrlAbstract/FREE Full Text

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • CT-S and CG are joint senior authors

    • Contributors AH and CTS carried out all screening of literature; AH and JJK extracted data; AH, JJK, CTS and CG interpreted results and drafted the manuscript.

    • Funding This work was supported by the award of a Capacity-Building Studentship to AH from the Medical Research Council (MRC) (grant number G1000397 – 1/1) North West Hub for Trials Methodology Research, UK (grant number G0800792).

    • Competing interests None.

    • Patient consent Obtained.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement Data extraction form and protocol available on request from ahoddy@liverpool.ac.uk.