Search strategy

This review followed the guidelines set forth by Aromataris and colleagues.16 We systematically searched through 12 different databases, including: Academic Search Premier, AUSPORT, CINAHL, Cochrane Library, ERIC, Health Source: Nursing/Academic Edition, MasterFILE Premier, PsycINFO, PubMed/MEDLINE, Scopus, SPORTDiscus and Web of Science. The databases were searched from the inception of indexing until 24 September 2018 using the following search syntax: caffeine AND (meta-an* OR ‘systematic review’) AND (exercise OR training OR muscle OR ‘physical performance’). The search syntax for each database is provided in online supplementary table S1. Quotation marks and the wildcard symbol were used to narrow down the search. In each full-text that was read, we also screened the reference list as a part of a secondary search. The search was carried out independently by two authors (JG and IG) to prevent any selection bias. The authors independently examined the titles, abstracts and, when applicable, the full-texts of the identified publications. On examination, the authors compared their lists of included and excluded papers; any disagreements were resolved by discussion and agreement between the authors.

Inclusion criteria

We included reviews coupled with a meta-analysis that examined the acute effects of caffeine ingestion on any exercise performance-related outcome. Both peer-reviewed and conference papers published in English or other languages were considered. Meta-analyses that included studies that combined caffeine with other ergogenic compounds, such as taurine, were excluded as they do not allow for the differentiation of the effects between the compounds. However, meta-analyses that included studies comparing caffeine and carbohydrate ingestion versus caffeine alone were included as long as the effect of caffeine could be isolated (ie, two solutions were given to the participants—one with caffeine and one without). As reported by the Participant-Intervention-Comparison-Outcome (PICO) process, the following criteria were followed:

Participants

Apparently healthy individuals of both sexes and all ages.

Interventions

Any acute study examining the effects of caffeine ingestion on exercise performance.

Comparison group

Placebo (provided that the effects of caffeine could be isolated).

Outcome measures

Any form of exercise performance.

Data extraction

The following data were extracted from the included meta-analyses: (1) the list of authors and year of publication; (2) the number and type of studies included in the meta-analysis; (3) the pooled number of participants; (4) the type of exercise test that was evaluated; (5) the pooled effect size with the 95% CI; (6) p values; and (7) percent changes and I² values. The same two authors that carried out searches also conducted the data extraction process. All data were tabulated to a spreadsheet predefined for this review. After data extraction, the spreadsheets were cross-checked between the authors for accuracy.

Methodological quality evaluation

The methodological quality of the included meta-analyses was assessed using the validated Assessing the Methodological Quality of Systematic Reviews 2 (AMSTAR 2) checklist.17 Two reviewers (JG and IG) independently assessed the methodological quality of the included reviews using the AMSTAR 2 checklist. This checklist contains 16 items that include questions regarding the use of the PICO description as a part of the inclusion criteria, the a priori registration of the review design, the comprehensiveness of the literature search, the number of authors that performed that search and data extraction, the description of included studies, the assessment of the quality of the included primary studies, reporting of sources of funding in the primary studies, the use of appropriate statistical methods, assessments of heterogeneity in the meta-analyses and reporting of the potential conflicts of interest. Full details on the checklist can be found in the paper by Shea et al.17 Each item on this checklist is answered with a ‘yes’, ‘no’, ‘cannot answer’ or ‘not applicable’. Out of these possible answers, only the ‘yes’ answer counts as a point in the total score for the assessed review. Based on the summary point scores, the meta-analyses were categorised as high quality (at least 80% of the items were satisfied), moderate quality (between 40% and 80% of the items were satisfied) or low quality (less than 40% of the items were satisfied), as performed previously.18 19

Quality of evidence

To assess the quality of evidence we used the modified Grading of Recommendations Assessment, Development and Evaluation (GRADE) principles.20 For the purpose of this review, we examined the following GRADE aspects: (1) risk of bias (determined by the quality of the primary studies, as assessed in the original reviews); (2) inconsistency (determined by variables such as the variation in the effects across the included studies and the overlap of the 95% CIs between the studies); (3) indirectness (determined by the generalisability of the findings while considering the study populations included in the primary research); (4) imprecision (determined by the total sample size in the analysis and the width of the 95% CI of the pooled effect size); and (5) publication bias (determined if the effect size of the largest study in each analysis was smaller than the pooled estimate from the meta-analysis and by examining the asymmetry of the funnel plot). Based on these criteria, the meta-analytical evidence was classified as high, moderate, low or very low. The GRADE assessment was conducted independently by two authors (JG and IG), with discussion and agreement for any differences.

Prediction interval (PI)

Using the number of included studies, the pooled standardised mean difference, the upper limit of the 95% CI and the tau-squared values (from each analysis), we calculated 95% PI for all included meta-analyses (spreadsheet available at: https://www.meta-analysis.com/pages/prediction.php). The 95% PI represents the range in which the effect size of a future study conducted on the topic will most likely lie. If the tau-squared values were not provided in the meta-analysis, these data were either requested from the authors or recalculated based on the data presented in the included studies.

Search results

The initial literature search identified 405 search records. Out of that pool of search results, 18 full-texts were read. Seven reviews were excluded after reading the full-texts.7 21–26 The reasons for their exclusion are provided in online supplementary table S2. Eleven reviews (with a total of 21 meta-analyses) were included in this umbrella review.4–6 8–15 All included reviews were published in peer-reviewed journals. The flow diagram of the search process can be found in figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Flow diagram of the search process.

Characteristics of the meta-analyses

The included meta-analyses were published between the years 2004 and 2018. The median number of studies included per meta-analysis was 19 (range: 2–44). The prevalence of primary studies with male-only samples ranged from 72% to 100% across the meta-analyses. The assessed outcomes in the meta-analyses included: maximal speed during running, cycling or rowing (defined as the maximal achieved speed in exercise performance tests lasting from 45 s to 8 min that had either a fixed duration or a fixed distance), aerobic endurance (assessed by time-to-exhaustion, time trial and graded exercise tests), peak and mean power in the 30 s Wingate test, peak torque in an isokinetic strength assessment, strength in the one repetition maximum (1RM) test, height in a vertical jump test, muscular endurance (assessed both using isometric and dynamic tests), duration of time trial or power during a time trial and maximal voluntary strength (assessed by pooling isometric, isokinetic and 1RM tests). A summary of the included meta-analyses can be found in table 1.

Effects of caffeine on exercise performance

The effects of caffeine ingestion on aerobic endurance were examined in five reviews with a total of nine meta-analyses; the majority reported ergogenic effects of caffeine (effect size range: 0.22–0.61). The range of included primary studies was from 2 to 44 (average: 23 studies). Doherty and Smith6 did not report significant effects of caffeine on aerobic endurance performance when considering only graded exercise tests and including six studies. Gonçalves Ribeiro et al 8 did not report significant effects of caffeine on this outcome (analysed using maximum running distance tests) while including two studies. The 95% PIs for these analyses are reported in table 1.

Four analyses examined the effects of caffeine on different measures of muscle strength. In three of these analyses, an ergogenic effect of caffeine was observed (effect size range: 0.16–0.20). The range of included studies was from 3 to 27 (average: 13 studies). In the analysis by Grgic and Pickering,10 the 95% PI was from −0.17 to 0.49. In the analysis by Grgic et al,11 the 95% PI was from 0.02 to 0.39, while in Warren et al’s15 analysis, the 95% PI was from −0.18 to 0.56. The 95% PI in the analysis by Polito et al 12 (this analysis did not report significant effects of caffeine on 1RM strength) was from −0.09 to 0.27.

Two analyses examined the effects of caffeine on muscular endurance. Both reported ergogenic effects of caffeine (effect size range: 0.28–0.38). Polito et al 12 included 16, while Warren et al 15 included 23 studies. The 95% PI was from 0.02 to 0.74 and from −0.29 to 0.85 for the analyses by Polito et al 12 and Warren et al,15 respectively.

Anaerobic power was examined in three analyses. In a meta-analysis including four studies, Gonçalves Ribeiro et al 8 did not report significant ergogenic effects of caffeine on power. The 95% PI in this analysis was from −0.65 to 1.01. In an analysis including 16 studies, Grgic9 reported ergogenic effects of caffeine on both mean and peak power (effect size range: 0.18 to 0.27). In the analysis for peak power, the 95% PI was from −0.35 to 0.89, while in the analysis for mean power, the 95% PI was from 0.04 to 0.32.

One meta-analysis, including 10 studies, examined the effects of caffeine on vertical jump height and reported an ergogenic effect of caffeine (effect size: 0.17).11 The 95% PI was from −0.03 to 0.37.

One meta-analysis, included nine studies, examined speed during running, cycling or rowing and reported ergogenic effects of caffeine (effect size: 0.41).4 The 95% PI was from 0.09 to 0.73.

One meta-analysis examined various forms of ‘short-term high-intensity exercise’ while pooling the effects of caffeine on: (1) time to exhaustion in various high-intensity short-term cycling and running efforts; (2) mean power, peak power output and total work during high-intensity short-term cycling; and (3) time trial time during 2000 m rowing.6 This analysis included 16 studies and reported ergogenic effects of caffeine of 0.16; the 95% PI was −0.18 to 0.50.

Besides the main analysis (presented in figure 2), several reviews also conducted additional subgroup analyses (eg, for trained vs untrained individuals, for upper vs lower body musculature) and these results are summarised in online supplementary table S3.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Summary of the effect sizes, 95% CIs (presented in the black lines), and 95% prediction intervals (95% PIs; presented in the grey lines) from the included meta-analyses. If there is no 95% PI presented, it was the same as the 95% CI.

Methodological quality evaluation

The methodological quality of the 11 included reviews is summarised in table 2. The reviews scored from 44% to 88% of the maximum 16 points. Three reviews were classified as being of high quality, while eight were classified as being of moderate methodological quality. None of the reviews were considered to be of poor methodological quality. Several criteria on AMSTAR 2 checklist were under-reported in the analysed reviews: (1) none provided an a priori design (ie, registration of the review methods in advance); (2) in four and five analyses the number of authors conducting the search and data extraction was not clear, respectively; (3) the list of excluded studies was not provided in any of the included reviews; and (4) sources of funding for the studies included in a given review were discussed only in three reviews.

Quality of the evidence

Based on the GRADE assessment, the included analyses were considered as providing very low (3 meta-analyses), low (7 meta-analyses) or moderate quality of evidence (11 meta-analyses). For risk of bias, several reviews did not assess the quality of the included studies and thus were given ‘unclear’ on this criterion. The meta-analyses were considered as not having serious inconsistency but were considered as having serious indirectness. The analyses were mostly considered as being ‘precise’ on the imprecision GRADE item. Finally, three meta-analyses were considered as ‘strongly suspected’ on the publication bias GRADE item. The results for each analysis are presented in online supplementary table S4.

Generalisability of the results

Based on the GRADE assessment for directness of evidence, the included reviews were rated as having serious indirectness given that the evidence is not direct enough to apply to all populations. Scrutiny of the meta-analyses included in this umbrella review highlights that primary studies conducted among women are lacking. Specifically, in all of the included meta-analyses, 72%–100% of the pooled sample participants were men. Women may metabolise caffeine differently than men given that changes in circulating steroid hormones during phases of the menstrual cycle can impact caffeine elimination in women,27 28 which might also impact the ergogenic effect of caffeine on exercise performance in this population. When conducting studies in women, the differences in caffeine metabolism across the follicular and luteal phase of the menstrual cycle may increase the complexity of the study design, which might partially explain why studies in this population are lacking. While there are studies conducted in both sexes that suggest both men and women may experience similar acute effects of caffeine ingestion on exercise performance,29 30 the generalisability of the meta-analytic findings is, however, limited mostly to men.

The majority of the primary studies were conducted in young individuals, and therefore, several meta-analyses are limited exclusively to young individuals.4 8 9 11–14 This may be relevant to highlight given that in animal models, with ageing, there appears to be a reduced ergogenic effect of caffeine.31 Caffeine has been shown to elicit positive effects on mood and cognitive performance in older adults.32 If caffeine also increases exercise performance in older adults, it might also enhance performance during activities of daily living in these individuals. This is particularly important from a public health point of view, given that reduced physical functioning (eg, in terms of reduced strength) may impact the quality of life in this population group.33 Although some of the studies conducted in older adults showed an ergogenic effect of caffeine on exercise performance,34 35 additional studies that directly compare the effects of caffeine between young versus older individuals are needed to explore if the effects of caffeine differ between age groups.

Methodological quality

While the meta-analyses included in the present umbrella review show that caffeine ingestion may indeed be ergogenic across a large range of exercise tasks, some additional considerations may help to improve future meta-analyses on this topic. Several of the included reviews did not adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, which currently represent a widely accepted standard for reporting meta-analyses. It should be taken into account that the PRISMA guidelines were published in 2009, which is 5 years after the review by Doherty and Smith.6 Nonetheless, several meta-analyses that did not follow the guidelines were published following the release of the PRISMA statement.4 5 8 15

None of the 11 reviewes registered their protocol for a review and thus did not receive a point on item 2 of the AMSTAR 2 checklist. Protocols of systematic reviews can be registered in the PROSPERO database. If registered, such protocols can help reduce the risk of wasteful duplication of reviews by independent research groups. However, the PROSPERO database is primarily focused on health outcomes and not exercise performance. As stated on their website, ‘PROSPERO includes protocol details for systematic reviews relevant to health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome’. The authors are not aware of any registries that focus on the publishing of protocols for systematic reviews in the sport and exercise field. Given that the number of published systematic reviews has increased over the last years, the formation of such a register for this line of research appears warranted.

Publication bias, as highlighted by Borenstein et al,36 can occur because studies that report higher (and significant) effect sizes are more likely to be published than those with low or non-significant effect sizes (ie, the file drawer problem). Therefore, the inclusion of only published studies in a given meta-analysis can lead to publication bias and may be a concern for the validity of the results. Four meta-analyses included in this umbrella review also examined unpublished literature in the form of master’s theses and doctoral dissertations.5 11 12 15 In the meta-analysis by Conger et al,5 the effect size of the unpublished studies was 0.13 (95% CI –0.08 to 0.33), while the effect size of the published studies was 0.32 (95% CI 0.19 to 0.46). These results might indeed suggest that studies with smaller effect sizes tend to remain unpublished and to avoid publication bias future meta-analyses should consider including unpublished results as well. The reviews that included unpublished literature highlight that, in many cases, such unpublished documents may be of equal or even greater methodological quality as those found in peer-reviewed journals. The influence of unpublished results can be examined by conducting a sensitivity analysis in which the pooled results are inspected after the exclusion of these studies. In this context, journal editors and reviewers are also encouraged to facilitate greater acceptance and publication of studies with results that would appear to be ‘less favourable’ (or statistically non-significant) to truly progress this area of work.

The spread of summary effects

Based on the GRADE assessment of inconsistency, the reviews were classified as not possessing serious inconsistencies. Indeed, the effect sizes across individual studies indicate that the studies rarely show a negative effect of caffeine supplementation on exercise performance. The effects in the primary studies were either positive or around the null value. In addition, the 95% CI from the primary studies largely overlap.

One interesting aspect refers to the spread of summary effects. Historically, caffeine ingestion has been suggested to predominantly provide a performance-enhancing effect on aerobic exercise performance.37 As shown both here and by others,38 39 it is evident that caffeine ingestion enhances performance in anaerobic exercise tasks as well. However, it is possible that the magnitude of the effect of caffeine is greater for aerobic as compared with anaerobic exercise. The effect sizes for meta-analyses that focused on aerobic tests of performance are generally higher than those that used anaerobic tests of performance (figure 2). Future studies may consider investigating the effects of caffeine ingestion on both aerobic and anaerobic tests of performance in the same sample to further explore whether the effect size magnitude differs between tasks that rely on predominantly oxidative or predominantly non-oxidative energy pathways.

The optimal dose of caffeine

While the included meta-analyses report that caffeine ingestion may be ergogenic across a broad range of exercise activities, the ‘optimal’ dose of caffeine remains elusive. If we observe the dosage used in the primary studies (across all of the included meta-analyses), it is clear that most of the studies used a single dose of caffeine (most commonly 6 mg/kg). Warren et al 15 examined the dose–response effects between the amount of caffeine ingested and its ergogenic effect on muscular endurance. This analysis found that for an increase in caffeine dose by 1 mg/kg, the relative effect size for muscular endurance increased by 0.10. However, these results should be interpreted with caution given that the dosage explained only 16% of the between-study variance. To explore the optimal doses of caffeine for exercise performance future dose–response studies are needed. The optimal doses may differ based on the source of caffeine,40 exercise test,41–44 muscle action type45 and between individuals,46 47 which needs to be taken into account when prescribing caffeine supplementation.

Is coffee a good way to consume caffeine?

While the results of this umbrella review suggest that caffeine is ergogenic in the majority of exercise situations, it is important to keep in mind that the majority of studies use caffeine anhydrous (highly concentrated caffeine powder) as the caffeine source, with a smaller group of studies utilising caffeine-containing supplements such as energy drinks, bars and gels. Coffee—the most widely used method of caffeine ingestion globally—is relatively underexplored as a pre-exercise performance enhancer. Hodgson and colleagues48 reported that caffeine anhydrous and coffee, standardised to deliver a caffeine dose of 5 mg/kg, were similarly effective in enhancing aerobic endurance performance. Similar results have been reported for resistance and sprint exercise.49 50 As a result, coffee is likely an effective ergogenic aid; the main issue here is a practical one. To be ergogenic, the caffeine dose from coffee likely has to fall within the 3–6 mg/kg range. The caffeine dose received from coffee depends on many factors, including bean type, preparation method and size of the cup; there are large differences in caffeine concentrations between different coffee brands and flavours and within the same brand across time.51–53 As a result, while the ‘average’ cup of coffee contains around 100 mg of caffeine—meaning that two cups would deliver ~200 mg, representing ~3 mg/kg for a 70 kg individual—this amount is hard to quantify in the specific cup of coffee that person is drinking.51 While keeping those caveats in mind, as a broad rule of thumb, two cups of coffee, consumed around 60 min before exercise, should exert an ergogenic effect in most individuals.

Suggestions for future research

Subgroup analyses conducted in the included meta-analyses in most cases are based on a low number of included studies (or effect sizes), which limits any definitive conclusions. Many areas remain unclear when it comes to caffeine supplementation. Some of these areas include: