Article Text

Download PDFPDF

Grey matters; on the importance of publication bias in systematic reviews
  1. Marinus Winters1,
  2. Adam Weir2,3
  1. 1Department of Rehabilitation, Nursing Science & Sports, University Medical Centre Utrecht, Utrecht, The Netherlands
  2. 2Aspetar Orthopaedic and Sports Medicine Hospital, Doha, Qatar
  3. 3Amsterdam Centre for Evidence Based Sports Medicine (ACES), Academic Medical Centre Amsterdam, Amsterdam, The Netherlands
  1. Correspondence to Dr Marinus Winters, Department of Rehabilitation, Nursing Science & Sports, University Medical Centre Utrecht, P.O. Box 85500, Utrecht 3508 GA, The Netherlands; marinuswinters{at}

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


‘How can I treat my patient best?’ is a question clinicians ask themselves on a daily basis. For clinical inquiries, systematic reviews (SRs) deliver the highest level of evidence in medical sciences.1 They can cover a variety of domains, including treatment and diagnostics. Presenting a full overview of the literature and delivering an unbiased estimate of effect are key aims of SRs. In this editorial, we explore publication bias, one of the potential sources of bias in SRs.

Publication bias: is a PubMed search enough?

Authors of SRs usually search multiple databases to find relevant publications. However, is all the available evidence identified with this approach? Publication bias is one threat to delivering an unbiased effect estimate.2 ,3 Publication bias is a phenomenon where journals are more likely to publish positive results, as these articles yield more citations. Non-significant publications are more prone to be rejected and remain unpublished.4 Unpublished resources are known as ‘grey literature’.

Publication bias has been shown in psychology research: Driessen et al5 checked all awarded US National Institutes of Health's grants for randomised controlled trials comparing psychological treatments to other interventions or to control groups. They examined if grants led to publication. Unpublished studies’ data were requested from the authors. Thirteen of 55 trials (24%) remained unpublished (11 with no significant effects), and another 2 trials never started. Adding unpublished studies’ results to published studies reduced the effect size for psychological treatment compared to control groups by 25%.5

In sports medicine, we are unaware of studies demonstrating publication bias altering effect estimates. Our team performed a SR on the treatment medial tibial stress syndrome. We performed a grey literature search, including a trial register search. Four of the 11 trials included were not published in international journals. These trials did not change effect estimates, but they contributed to improved estimation precision. Furthermore, two studies reported on interventions that were not investigated in the published reports.6

Page et al7 performed a cross-sectional study in which they assessed the epidemiological and reporting characteristics of a random sample of 300 SRs indexed in MEDLINE in February 2014. Only 7% of the SRs sought for unpublished studies. This suggests that publication bias is poorly dealt with in SRs in general. To assess whether SRs in BJSM search for grey literature, we assessed all SRs published in BJSM from January to July 2016 (issues 1–12).

How do BJSM SRs deal with publication bias?

We identified 20 SRs published between January and July 2016, and 15 (75%) searched for unpublished studies. Most of those studies (67%) searched a single grey literature database only. Fifty per cent of the SRs searched SPORTDiscus, 20% Web of Science and 20% Google Scholar (20%). One SR searched Biological Abstracts, and one SR examined conference proceedings from two annual conferences specific to the SR's topic, for the past 10 years. One study sought contact with experts for unpublished material. None of the reviews searched in other grey literature databases such as Open Grey, The British Library Inside or BIOSIS Previews. Only one SR searched trial registers; however, they searched in four national registers only, which probably does not cover all registered trials. An overview of grey literature searches in the identified SRs can be found online.

Supplementary material

How can we best deal with publication bias?

To control for publication bias is important, knowing that it is unlikely that publication bias can ever be fully addressed (we do not know about the studies that are shelved and hidden; the file draw phenomenon). There are multiple ways to control for publication bias: (1) trials need to be registered before study initiation, (2) funnel plots should be used to visually assess the presence of publication bias and (3) multiple grey literature sources should be searched with a sensitive search strategy. In most countries, clinical trials must be listed in trial registers. Furthermore, studies are often presented at conferences before they are published, allowing them to be identified. There are several resources for grey literature in sports medicine:

  • Web of Science;

  • BIOSIS Previews;

  • SPORTDiscus;

  • Google Scholar;


  • British Library Inside.

This list is not exhaustive and there could be other good resources, depending on the topic of the SR.

Second, international trial registries offer great opportunities to detect those studies that have remained unpublished or are ongoing. The following international trial registries seem most relevant, as they cover a great amount of national trial registers:

  • WHO International Clinical Trials Registry Platform;

  • US National Institutes of Health Trial Register;

  • European Clinical Trial Register;

  • ISRCTN registry.

Another way to find unpublished or ongoing studies is to contact fellow experts.8


Unpublished studies can substantially change effect estimates in SRs. We urge SR authors to include an extensive grey literature search to enable the delivery of unbiased estimates of effects in SRs.



  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.