Article Text

Download PDFPDF

Next-generation systematic reviews: prospective meta-analysis, individual-level data, networks and umbrella reviews
  1. John Ioannidis
  1. Correspondence to Dr John Ioannidis, Stanford Prevention Research Center, Department of Medicine, and Department of Health Research and Policy, Stanford University School of Medicine and Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, and Meta-Research Innovation Center at Stanford (METRICS), Stanford, California, USA; jioannid{at}stanford.edu

Statistics from Altmetric.com

Systematic reviews and meta-analyses have reached a critical point. Their success is solid and dreadful at the same time. They are widely considered the highest level of evidence. There are tens of thousands of systematic reviews already published, but their production is still increasing geometrically. The problem is that the majority of systematic reviews are flawed, misleading, redundant, useless or all of the above,1 and this applies to almost all medical fields, including sports and exercise medicine.

Part of the problem with systematic reviews stems from the poor, misleading primary evidence2 that authors try to synthesise and make sense of. However, if this were the only major problem, systematic reviews and meta-analyses would still be extremely useful. They could focus exactly on showing how flawed, misleading and useless this evidence is. This could lead to suggestions on how to improve research in the field. Instead, systematic reviews often sanctify results from poor studies, by making these seem even more statistically significant and (spuriously) conclusive. They can also compound the problems seen in the primary studies, if they are driven by reviewers and sponsors with conflicts of interest, financial or academic.

There are four types of ‘next-generation’ systematic reviews that may raise the bar and help shape a new generation of more reliable evidence synthesis: prospective meta-analysis, individual-level data, network meta-analyses and umbrella reviews. They are not necessarily brand new ideas, but in the current circumstances of uncontrollable overproduction and unchecked quality, they have a fresh opportunity for impact. This large-scale impact would have been unimaginable in the past due to constrains in access to data, limited availability of sophisticated methods and fewer opportunities for their application. None of these next-generation tools are bullet proof, but they hold promise and some distinct advantages. Their disadvantages also need to be recognised. This piece outlines how these next-generation approaches might help improve the quality of evidence synthesis in sports and exercise medicine and associated disciplines.

Prospective meta-analysis

This approach3 refers to the design of multiple trials with the explicit, predefined purpose to, when completed, combine them in a meta-analysis. Depending on the breadth of the questions answered, prospective, preplanned meta-analyses can either address focused questions or shape large research agendas for a long-term research programme or even an entire discipline.4 A broad agenda approach would be ideal, but it usually requires wider buy-in from multiple stakeholders who operate in the field, conducting and funding its research. However, even if not all trials and all stakeholders can participate, a prospective meta-analysis is still useful. Key benefits of prospective meta-analyses are that designs have been thoroughly vetted among colleagues and preregistered, preplanned analyses will be performed with widely scrutinised, prespecified high-quality standards, and there will be no selective reporting of the results.

Prospective meta-analyses may encompass trials that use different designs and test a variety of interventions, comparisons, settings and populations, provided it is all transparent and preconceived. Thus, prospective meta-analysis is a design ideally suited when there are different interventions and different candidate modes to implement them, for example, exercise training regimens. It is not surprising that the first major prospective meta-analysis involved trials of diverse exercise interventions for reducing falls in elderly people.3 Prospective meta-analyses have nevertheless been very uncommon to date, since they require extensive co-ordination. Funders and professional societies may wish to rekindle interest in them. This design may be useful to consider in regulatory decisions, for example, in monitoring evidence for adverse events.5 Licensing of prescription medicines and medical devices is currently using some set of trials whose conduct is presubmitted to regulatory authorities. Their consistent prospective integration in a formal meta-analysis plan is a natural step.

Meta-analysis of individual patient/athlete/participant data

This meta-analytic approach appeared in the 1990s, along with prospective meta-analysis.6 Meta-analyses of individual data have been more popular of the two approaches, with close to a thousand such meta-analyses done to date.7 Nevertheless, they still represent a small minority within the bulk production of systematic reviews. Until recently, it has typically required a lot of effort to collect the raw data from multiple trials, and the limited range of information collected has allowed for relatively few additional analysis options beyond what summary data can offer. Moreover, some investigators may not cooperate to contribute data. These limitations put a cap to the potential advantages of cleaning, harmonising variables across studies, correcting and extending (eg, longer follow-up) data and addressing effect modification in various patient subgroups.8 Until now, selective reporting bias was probably just as big a problem for individual-level meta-analyses as it is for summary meta-analyses, because unavailable trials are difficult to unearth with either approach. Individual-level data meta-analyses are less ‘systematic’ in the sense that they may eventually retrieve even fewer trials with raw data than published trials with available summary results.

Applications of meta-analyses of individual-level data in randomised trials of sports and exercise medicine to date have been few, and typically have included limited total sample sizes of a few hundred participants.9–15 This has led to no major successes in individualising or at least stratifying treatment. Two such analyses showed that exercise can decrease cigarette craving,9 10 but no moderators of the treatment effect were identified.

Applications of individual-level meta-analyses in the field have been probably more popular and successful in projects involving synthesis of observational data.16–18 For example, a meta-analysis of over 1 million people in prospective cohorts evaluated associations of physical activity and sitting behaviour with death risk16; another meta-analysis addressed the association between physical activity and cancer risk17; and another one combined data from multiple surveys on how physical activity modulates the association between alcohol and mortality.18 Meta-analyses of observational data carry all the major threats of observational evidence, in particular confounding. The availability of detailed, harmonised individual-level data may allow occasionally better handling of confounding. Selection bias is an even greater threat for observational data than for randomised trials. Selection bias may be even larger in meta-analyses of observational individual-level data (eg, investigators may seek to include primarily studies with results that conform to their expectations).

All things considered, individual-level data meta-analyses are likely to witness a renowned interest in the near future. After many obstacles, obtaining raw data from clinical trials and other clinical research seems at last to become a reality.19 20 This means that the cost, effort and frustration accompanying these individual-level meta-analyses will hopefully decrease substantially. Available data may be richer and more comprehensive to allow meaningful assessment of effect modification—an important step forward.

Network meta-analysis

This approach is another way to enhance the breadth of evidence synthesis.21 Instead of focusing on a narrow question with a single treatment comparison, a network meta-analysis examines all treatments for a given condition or disease and all the possible comparisons between them. This allows the analysis to draw strength from both direct and indirect evidence, and to assess the relative merits of all treatments of interest. Network meta-analyses may also build on prospective meta-analysis designs and on individual-level data, although the vast majority of them to date have built on summary data retrospectively extracted from published trials.

Network meta-analyses are relatively uncommon in sports and exercise medicine.22–28 They include networks that compare exercise versus drug treatments for mortality in various conditions,22 different physical activity and other interventions for osteoarthritis,23 24 and different interventions for knee arthroplasty25 lateral elbow tendinopathy and epicondylalgia26 27 and plantar fasciitis.28 Network meta-analyses are still susceptible to reporting biases and the other biases that traditional pairwise meta-analyses may suffer from. However, a broader picture of the evidence may allow a more balanced view of treatment effects, plus a broader picture of biases that may affect a given field.

Syste matic reviews of systematic reviews (umbrella reviews)

With tens of thousands of systematic reviews and meta-analyses being available, in most cases performing yet another one of the same topic may not be a high priority. Instead, it may be very useful to perform systematic reviews of systematic reviews, aka umbrella reviews,29 synthesising information from all systematic reviews and meta-analyses on a given topic. These umbrella reviews may be based on outcomes, risk factors or interventions.

One umbrella review of relevance to sports and exercise medicine evaluated 248 meta-analyses on 23 foods, 31 nutrients, eight indices of body size and three indices of physical activity (risk factors, some of which may also be translated into potential interventions) in association with risk of total prostate cancer development, mortality or cancer development by stage and grade (outcomes).30 There are <100 umbrella reviews performed to date, but their numbers may increase fast, since their content is an attractive way to distil and translate large amounts of evidence. Umbrella reviews allow a higher-level synthesis of the evidence and a better recognition of the uncertainties, biases and knowledge gaps. A bird’s eye view may allow understanding of the spread of summary effects, heterogeneity, hints of bias and quality features that affect the credibility of the results in different systematic reviews in a whole field.

Improving the quality of systematic reviews

Despite their weaknesses, systematic reviews and meta-analyses will continue to be extremely influential. As we become increasingly experienced in the strengths and limitations of these methods, we must try to incorporate the broader, next-generation evidence synthesis options more routinely. All four systematic review tools described above typically involve a lot more work and ambition than the ‘average’ systematic review currently published (that is invariably narrow in scope, with very narrow outcomes, includes very specific designs and addresses a very specific question). With the wider adoption of evidence synthesis methods, new opportunities and new threats emerge.

Systematic reviews are undertaken by a wide spectrum of diverse stakeholders, with a broad range of expertise (or lack of expertise) in the methods, and with variable protection from biases and conflicts of interest. Probably thousands of systematic reviews, including hundreds of sophisticated network meta-analyses, are currently conducted by contractor companies, hired mostly by industry.31 Much of their work is not published and it is possible that what gets published is biased in favour of the sponsor.32 Academic or commercial conflicts of interest may affect any systematic review, no matter how sophisticated.

But there is hope. The proposed next-generation tools should hopefully offer some better options and more transparency. All of them may offer feedback with suggestions on how to improve the design, quality and rigour of future primary studies. Eventually, primary studies may become largely synonymous with meta-analyses. For example, at some point, primary studies informed by filling a gap in research identified by an umbrella review may become part of prospective network meta-analyses using individual-level data.

References

View Abstract

Footnotes

  • Competing interests None declared.

  • Provenance and peer review Commissioned; internally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles