Publication bias: is a PubMed search enough?

Authors of SRs usually search multiple databases to find relevant publications. However, is all the available evidence identified with this approach? Publication bias is one threat to delivering an unbiased effect estimate.2 ,3 Publication bias is a phenomenon where journals are more likely to publish positive results, as these articles yield more citations. Non-significant publications are more prone to be rejected and remain unpublished.4 Unpublished resources are known as ‘grey literature’.

Publication bias has been shown in psychology research: Driessen et al5 checked all awarded US National Institutes of Health's grants for randomised controlled trials comparing psychological treatments to other interventions or to control groups. They examined if grants led to publication. Unpublished studies’ data were requested from the authors. Thirteen of 55 trials (24%) remained unpublished (11 with no significant effects), and another 2 trials never started. Adding unpublished studies’ results to published studies reduced the effect size for psychological treatment compared to control groups by 25%.5

In sports medicine, we are unaware of studies demonstrating publication bias altering effect estimates. Our team performed a SR on the treatment medial tibial stress syndrome. We performed a grey literature search, including a trial register search. Four of the 11 trials included were not published in international journals. These trials did not change effect estimates, but they contributed to improved estimation precision. Furthermore, two studies reported on interventions that were not investigated in the published reports.6

Page et al7 performed a cross-sectional study in which they assessed the epidemiological and reporting characteristics of a random sample of 300 SRs indexed in MEDLINE in February 2014. Only 7% of the SRs sought for unpublished studies. This suggests that publication bias is poorly dealt with in SRs in general. To assess whether SRs in BJSM search for grey literature, we assessed all SRs published in BJSM from January to July 2016 (issues 1–12).

How do BJSM SRs deal with publication bias?

We identified 20 SRs published between January and July 2016, and 15 (75%) searched for unpublished studies. Most of those studies (67%) searched a single grey literature database only. Fifty per cent of the SRs searched SPORTDiscus, 20% Web of Science and 20% Google Scholar (20%). One SR searched Biological Abstracts, and one SR examined conference proceedings from two annual conferences specific to the SR's topic, for the past 10 years. One study sought contact with experts for unpublished material. None of the reviews searched in other grey literature databases such as Open Grey, The British Library Inside or BIOSIS Previews. Only one SR searched trial registers; however, they searched in four national registers only, which probably does not cover all registered trials. An overview of grey literature searches in the identified SRs can be found online.

How can we best deal with publication bias?

To control for publication bias is important, knowing that it is unlikely that publication bias can ever be fully addressed (we do not know about the studies that are shelved and hidden; the file draw phenomenon). There are multiple ways to control for publication bias: (1) trials need to be registered before study initiation, (2) funnel plots should be used to visually assess the presence of publication bias and (3) multiple grey literature sources should be searched with a sensitive search strategy. In most countries, clinical trials must be listed in trial registers. Furthermore, studies are often presented at conferences before they are published, allowing them to be identified. There are several resources for grey literature in sports medicine:

• Web of Science;

• BIOSIS Previews;

• SPORTDiscus;

• Google Scholar;

• Opengrey.eu;

• British Library Inside.

This list is not exhaustive and there could be other good resources, depending on the topic of the SR.

Second, international trial registries offer great opportunities to detect those studies that have remained unpublished or are ongoing. The following international trial registries seem most relevant, as they cover a great amount of national trial registers:

• WHO International Clinical Trials Registry Platform;

• US National Institutes of Health Trial Register;

• European Clinical Trial Register;

• ISRCTN registry.

Another way to find unpublished or ongoing studies is to contact fellow experts.8