Meta-analysis of individual patient/athlete/participant data

This meta-analytic approach appeared in the 1990s, along with prospective meta-analysis.6 Meta-analyses of individual data have been more popular of the two approaches, with close to a thousand such meta-analyses done to date.7 Nevertheless, they still represent a small minority within the bulk production of systematic reviews. Until recently, it has typically required a lot of effort to collect the raw data from multiple trials, and the limited range of information collected has allowed for relatively few additional analysis options beyond what summary data can offer. Moreover, some investigators may not cooperate to contribute data. These limitations put a cap to the potential advantages of cleaning, harmonising variables across studies, correcting and extending (eg, longer follow-up) data and addressing effect modification in various patient subgroups.8 Until now, selective reporting bias was probably just as big a problem for individual-level meta-analyses as it is for summary meta-analyses, because unavailable trials are difficult to unearth with either approach. Individual-level data meta-analyses are less ‘systematic’ in the sense that they may eventually retrieve even fewer trials with raw data than published trials with available summary results.

Applications of meta-analyses of individual-level data in randomised trials of sports and exercise medicine to date have been few, and typically have included limited total sample sizes of a few hundred participants.9–15 This has led to no major successes in individualising or at least stratifying treatment. Two such analyses showed that exercise can decrease cigarette craving,9 10 but no moderators of the treatment effect were identified.

Applications of individual-level meta-analyses in the field have been probably more popular and successful in projects involving synthesis of observational data.16–18 For example, a meta-analysis of over 1 million people in prospective cohorts evaluated associations of physical activity and sitting behaviour with death risk16; another meta-analysis addressed the association between physical activity and cancer risk17; and another one combined data from multiple surveys on how physical activity modulates the association between alcohol and mortality.18 Meta-analyses of observational data carry all the major threats of observational evidence, in particular confounding. The availability of detailed, harmonised individual-level data may allow occasionally better handling of confounding. Selection bias is an even greater threat for observational data than for randomised trials. Selection bias may be even larger in meta-analyses of observational individual-level data (eg, investigators may seek to include primarily studies with results that conform to their expectations).

All things considered, individual-level data meta-analyses are likely to witness a renowned interest in the near future. After many obstacles, obtaining raw data from clinical trials and other clinical research seems at last to become a reality.19 20 This means that the cost, effort and frustration accompanying these individual-level meta-analyses will hopefully decrease substantially. Available data may be richer and more comprehensive to allow meaningful assessment of effect modification—an important step forward.

Network meta-analysis

This approach is another way to enhance the breadth of evidence synthesis.21 Instead of focusing on a narrow question with a single treatment comparison, a network meta-analysis examines all treatments for a given condition or disease and all the possible comparisons between them. This allows the analysis to draw strength from both direct and indirect evidence, and to assess the relative merits of all treatments of interest. Network meta-analyses may also build on prospective meta-analysis designs and on individual-level data, although the vast majority of them to date have built on summary data retrospectively extracted from published trials.

Network meta-analyses are relatively uncommon in sports and exercise medicine.22–28 They include networks that compare exercise versus drug treatments for mortality in various conditions,22 different physical activity and other interventions for osteoarthritis,23 24 and different interventions for knee arthroplasty25 lateral elbow tendinopathy and epicondylalgia26 27 and plantar fasciitis.28 Network meta-analyses are still susceptible to reporting biases and the other biases that traditional pairwise meta-analyses may suffer from. However, a broader picture of the evidence may allow a more balanced view of treatment effects, plus a broader picture of biases that may affect a given field.

Syste matic reviews of systematic reviews (umbrella reviews)

With tens of thousands of systematic reviews and meta-analyses being available, in most cases performing yet another one of the same topic may not be a high priority. Instead, it may be very useful to perform systematic reviews of systematic reviews, aka umbrella reviews,29 synthesising information from all systematic reviews and meta-analyses on a given topic. These umbrella reviews may be based on outcomes, risk factors or interventions.

One umbrella review of relevance to sports and exercise medicine evaluated 248 meta-analyses on 23 foods, 31 nutrients, eight indices of body size and three indices of physical activity (risk factors, some of which may also be translated into potential interventions) in association with risk of total prostate cancer development, mortality or cancer development by stage and grade (outcomes).30 There are <100 umbrella reviews performed to date, but their numbers may increase fast, since their content is an attractive way to distil and translate large amounts of evidence. Umbrella reviews allow a higher-level synthesis of the evidence and a better recognition of the uncertainties, biases and knowledge gaps. A bird’s eye view may allow understanding of the spread of summary effects, heterogeneity, hints of bias and quality features that affect the credibility of the results in different systematic reviews in a whole field.

Improving the quality of systematic reviews

Despite their weaknesses, systematic reviews and meta-analyses will continue to be extremely influential. As we become increasingly experienced in the strengths and limitations of these methods, we must try to incorporate the broader, next-generation evidence synthesis options more routinely. All four systematic review tools described above typically involve a lot more work and ambition than the ‘average’ systematic review currently published (that is invariably narrow in scope, with very narrow outcomes, includes very specific designs and addresses a very specific question). With the wider adoption of evidence synthesis methods, new opportunities and new threats emerge.

Systematic reviews are undertaken by a wide spectrum of diverse stakeholders, with a broad range of expertise (or lack of expertise) in the methods, and with variable protection from biases and conflicts of interest. Probably thousands of systematic reviews, including hundreds of sophisticated network meta-analyses, are currently conducted by contractor companies, hired mostly by industry.31 Much of their work is not published and it is possible that what gets published is biased in favour of the sponsor.32 Academic or commercial conflicts of interest may affect any systematic review, no matter how sophisticated.

But there is hope. The proposed next-generation tools should hopefully offer some better options and more transparency. All of them may offer feedback with suggestions on how to improve the design, quality and rigour of future primary studies. Eventually, primary studies may become largely synonymous with meta-analyses. For example, at some point, primary studies informed by filling a gap in research identified by an umbrella review may become part of prospective network meta-analyses using individual-level data.