You are here

Article Text

Article menu

Download PDFPDF

XML
Education review
Call to increase statistical collaboration in sports science, sport and exercise medicine and sports physiotherapy
  1. Kristin L Sainani1,
  2. David N Borg2,
  3. Aaron R Caldwell3,
  4. Michael L Butson4,
  5. Matthew S Tenan5,
  6. Andrew J Vickers6,
  7. Andrew D Vigotsky7,
  8. John Warmenhoven8,9,
  9. Robert Nguyen10,
  10. Keith R. Lohse11,
  11. Emma J Knight12,
  12. Norma Bargary13
  1. 1 Epidemiology and Population Health, Stanford University, Stanford, California, USA
  2. 2 Menzies Health Institute Queensland, Griffith University, Nathan, Queensland, Australia
  3. 3 Thermal and Mountain Medicine Division, US Army Research Institute of Environmental Medicine, Natick, Massachusetts, USA
  4. 4 Deptartment of Health & Medical Sciences, Swinburne University of Technology, Hawthorn, Victoria, Australia
  5. 5 Optimum Performance Analytics Associates LLC, Apex, North Carolina, USA
  6. 6 Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
  7. 7 Departments of Biomedical Engineering and Statistics, Northwestern University, Evanston, Illinois, USA
  8. 8 Exercise & Sport Science, Faculty of Health Sciences, University of Sydney, Sydney, New South Wales, Australia
  9. 9 Australian Institute of Sport, Canberra, Australian Capital Territory, Australia
  10. 10 Department of Mathematics and Statistics, University of New South Wales, Sydney, New South Wales, Australia
  11. 11 Health, Kinesiology, and Recreation; Department of Physical Therapy and Athletic Training, University of Utah Health, Salt Lake City, Utah, USA
  12. 12 School of Public Health, University of Adelaide, Adelaide, South Australia, Australia
  13. 13 Department of Mathematics and Statistics, University of Limerick, Limerick, Ireland
  1. Correspondence to Dr Kristin L Sainani, Epidemiology and Population Health, Stanford University, Stanford, California, USA; kcobb{at}stanford.edu

https://doi.org/10.1136/bjsports-2020-102607

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Statistical errors are common in many biomedical fields.1–5 We believe the nature and impact of these errors to be great enough in sports science and medicine to warrant special attention.6–14 Poor methodological and statistical practices have led to calls for change in other fields, such as psychology.15–18 We believe that a similar call to action is needed in sports science and medicine. Specifically, we see two pressing needs: (1) to increase collaboration between researchers and statisticians, and (2) to increase statistical training within the exercise science/medicine/physiotherapy (PT) discipline. Our call to action extends the work of those who have previously called for increased statistical collaboration in sports medicine and sports injury research.19–21

    Though some academic sports science and medicine studies employ statisticians, such collaborations are an exception rather than the norm. To determine the extent of collaboration, we performed a systematic review of articles published in quartile one sports science journals in 2019 (see online supplementary file 1 for methods and online supplementary file 2 for data). The initial extraction included 8970 articles; of the 400 articles selected at random, 299 were deemed eligible and included in the review (figure 1). We found that only 13.3% (95% CI: 9.5% to 17.2%) of papers had at least one coauthor affiliated with a biostatistics, statistics, data science, data analytics, epidemiology, maths, computer science or economics department (figure 2). It should be noted that we included a broad set of methodological departments because we recognise that individuals from these fields may possess considerable statistical expertise. When we use the term ‘statistician’ in this paper, we broadly include individuals from other methods-focussed disciplines if they have extensive statistical training and experience.

    Supplemental material

    Supplemental material

    Figure 1
    Figure 1

    Flowchart of the article search and inclusion for the systematic review.

    Figure 2
    Figure 2

    Percentage of data-containing articles in quartile one sports science journals that include at least one coauthor affiliated with a statistics or other methodologically-oriented department (from our systematic review, n=299). Statistics includes biostatistics, statistics, data science and data analytics departments; epidemiology includes authors from departments of community health, population health, health or public health if they are trained as epidemiologists or statisticians; computer science includes information technology department.

    The shortage of statisticians working in the field means that sports science and medicine researchers are often designing studies and running analyses by themselves. Some of these researchers undertake in-depth training in statistics and are well-equipped to handle these tasks. However—as with other applied disciplines—sports science and medicine researchers often lack adequate training in study design and statistics, which can lead to errors.22–24 This is especially problematic as study designs and data sets become more complex.

    We are also concerned by a phenomenon in sports science and medicine. Scientists in these fields are developing statistical methods and introducing them into the literature without adequate peer review from the statistics community.25–27 Many of these methods are statistically and mathematically flawed.28 29 While advances in statistics sometimes have come from applied disciplines (eg, work on measurement done in education and psychology), these novel statistical methods were presented, critiqued and evaluated in the statistical literature before they were introduced and used in an applied context.

    In this commentary, we present two series of case studies that illustrate the importance of effective collaboration between sports science and medicine researchers, and statisticians. We discuss barriers that have prevented collaboration. We recommend next steps forward.

    Case studies: avoidable statistical errors

    Statistical errors can occur during study design, data analysis or reporting. The case studies described below do not provide an exhaustive list of possible errors. Rather, we highlight several instances where an error may have been avoided with more statistical knowledge or greater collaboration with statisticians. Other references provide further examples of common statistical errors in sports science and medicine.6–11

    Errors in study design—exercise physiology

    A study of 14 active men aimed to establish the reliability of a biomarker test used to measure gastrointestinal (GI) integrity during conditions of heat stress.30 Participants performed two intermittent exertional heat stress tests, and GI integrity was measured with several biomarker tests, including the intestinal fatty-acid-binding protein (I-FABP). Authors reported that the I-FABP test at rest ‘displayed moderate-to-strong relative and acceptable absolute reliability between repetitions.’ However, this was based on finding significant correlation between the repeat measurements, with a Pearson correlation coefficient of 0.75 (p<0.01).

    This case illustrates two issues: (1) An intraclass correlation coefficient (ICC) is a more appropriate measure of reliability and (2) when we extracted data from figure 3 of that paper to roughly estimate the ICC (2,1), we found a value and 95% CI of 0.72 (0.32 to 0.89). This estimate is too imprecise to draw useful conclusions; reliability may plausibly be anywhere from insufficient to excellent. In this case, the authors failed to perform an a priori sample size calculation, leading to a study that was too small to adequately answer the question of interest.

    Errors in data analysis—nutrition and endocrinology in exercise

    A study of vitamin D levels and menstrual status in 77 college-aged women concluded that ‘Women who did not meet the recommended level of 30 ng/mL of 25(OH)D had almost five times the odds of having menstrual cycle disorders as women who were above the recommended vitamin D level’.31 The study has important implications for women athletes, who frequently experience menstrual cycle irregularities.

    A closer inspection of the analysis revealed problems. While a higher proportion of the low vitamin D group (40% of 60) had menstrual disturbances compared with the high vitamin D group (12% of 17), the analysis failed to account for important differences between the groups. The low vitamin D group was 17% heavier than the high vitamin D group—average body mass of 66.7 vs 57.0 kg. Body mass was also strongly related to menstrual disturbances: Women with menstrual disturbances had an average body mass of 77.6 vs 57.9 kg in women without menstrual disturbances. Thus, the apparent relationship between low vitamin D and menstrual disturbances may be caused entirely by strong confounding by body mass. The authors should have undertaken a multivariable analysis that accounted for body mass.

    Errors in statistical reporting—sports medicine/orthopedics/rehabilitation

    A large study was undertaken to understand factors that predict athlete recovery 2 years after an ACL reconstruction.32 The manuscript reports that: ‘Multivariable regression analyses were constructed to examine which baseline risk factors were independently associated with each outcome variable…primary outcome variables were all treated as continuous’, but the manuscript reports ORs. ORs are typically reported for binary, not continuous outcomes. This discrepancy caught the eye of an author in the present commentary, and a series of letters to the editor33 34 determined that a highly nuanced, thoughtful and appropriate analysis was performed on the data. However, the modelling approach was poorly described—which makes it difficult to judge the validity of the study and also hampers reproducibility. In this case, the research team included individuals with statistical expertise who were involved in study planning and data analysis; however, these individuals may have been insufficiently involved in drafting the paper.

    Case studies: inventing new statistics

    Introducing new statistical methods into the literature typically involves several steps: (1) writing down mathematical equations that explicitly formulate the method; (2) establishing the empirical behaviour of the method through mathematical proofs, simulations or both; and (3) publishing in a statistics journal or in a general interest journal following peer review by statisticians. Given the technical expertise required, statisticians or mathematicians are integral to the process.

    A classic example is the significance analysis of microarrays (or SAM) statistical technique, which was introduced in 2001.35 SAM arose from a collaboration between the statistician Robert Tibshirani and biologists Virginia Goss Tusher and Gilbert Chu, who were trying to develop better ways to analyse microarray data. The initial paper on SAM was published in PNAS (Proceedings of the National Academy of Sciences of the United States of America); contains mathematical equations and proofs; and formally compares the performance of SAM to other methods that were popular for analysing microarray data at that time. This is one example; statistical journals publish numerous papers each year introducing new statistical approaches. Here, we highlight three cases where statistical methods were introduced into the literature without proper statistical vetting.

    Methods for identifying responders and non-responders

    Sports science and medicine researchers are interested in identifying ‘Responders’ and ‘Non-responders’ to exercise interventions. While response heterogeneity has been covered at great length in the applied statistics literature,36 these guidelines have largely been overlooked in sports science and medicine. Authors in these fields have employed a variety of analytical techniques for identifying differential response, including k-means cluster analysis followed by analysis of variance (ANOVA),37 grouping response based on the SE of measurement,38 and more recently, a novel analytical algorithm was suggested.27 However, none of these approaches are statistically or philosophically grounded, and indeed, they have poor statistical properties, such as high Type I error rates.29 39

    Modifications of principal components analysis

    In another example,26 modifications to the application of principal components analysis (PCA) applied within functional data analysis40 were proposed for the context of exploring high-dimensional kinematic sports science data.41–44 PCA estimates the principal components of a set of curves whose measured values are stored in a data matrix such that each row holds the data for an individual curve. In a recent pre-print on SportRxiv,26 the author argues that estimating the principal components of the data matrix, such that each column holds the measured values of an individual curve, is more appropriate. However, this alternative approach violates the independence assumption of PCA, does not centre the data conventionally, interprets the resulting scores as loadings, and has been criticised by an expert in the field.45 Such modifications to techniques like PCA should be carefully reviewed by the statistics community before being promoted as more appropriate than their conventional application. This is necessary to avoid confusion in the application of PCA by scientists in applied fields.

    Magnitude-based inference

    The case of Magnitude-Based Inference (MBI) is a cautionary tale of what can happen when a novel statistical approach is widely adopted before being vetted. MBI appeared in the sports science literature in 200625 in a way that was highly unusual for a statistical method. The introductory paper contained no mathematical formulas and no mathematical proofs or simulations demonstrating the method’s empirical behaviour. The paper was published as a commentary in the sports science literature, not a methodological journal, and it was not peer-reviewed by statisticians. The method has been criticised by the statistics community (and authors of this paper) for over a decade.28 46–52 In addition to lacking a sound mathematical foundation,28 46 51 the method leads to high Type I error rates that are not transparent25 37 38 42 and frequently leads researchers to reach overly optimistic conclusions.48 52 These critiques of MBI have even garnered negative attention for sports science in the popular media.12–14

    In all these cases, effective collaboration with statisticians could have: (i) pointed researchers to existing methods that accomplish the same analytical goals or (ii) helped researchers to mathematically formalise new methods and assess their statistical properties. Indeed, theoretical breakthroughs are often inspired by practical needs.

    Barriers to collaboration

    The numerous barriers to collaboration between statisticians and sports scientists are comparable to those that hinder collaboration between statisticians and many other applied disciplines. Universities and research institutes are often spatially organised by discipline, which offers little opportunity for sports scientists and statisticians to interact.53 Many scientific disciplines employ intermediate methodological specialists to help bridge this gap—for example, psychology has mathematical psychologists and chemistry has instrumental chemists. Unfortunately, methodological specialists are less common in sports science. Sports analytics is a rapidly growing sub-discipline of sports science,19 but most sports analysts are currently employed by professional sports teams and have a narrow focus on performance metrics; sports scientists with a high level of statistical training are lacking in academia.

    Sports science and medicine researchers bring subject matter expertise across a range of disciplines, including physiology, biomechanics, nutrition and psychology. However, some of these researchers may only receive limited training in study design and statistics. Like many applied researchers, they may try to learn applied statistics through self-study or from statistical ‘cookbook’ guides,22–24 and may fail to sufficiently appreciate the complexity of statistics and the in-depth expertise that statisticians, or those pursuing a formal statistics education, can bring to the table.54 The problem may be compounded when poor statistical techniques are passed from mentors to students, thus propagating poor practices to the next generation of researchers.55 Finally, a lack of statistical expertise in the journal peer review process means that many papers are published using suboptimal statistical methods;52 this often creates a positive feedback loop as methods are copied from paper to paper. BJSM (British Journal of Sports Medicine) only began having statistical Deputy Editors in 2019.

    Though many sports science and medicine researchers would welcome statistical support for their projects, such support is often unavailable. Mathematical statisticians receive greater academic recognition for theoretical than applied work, and thus may be uninterested in collaborating with applied researchers.56 57 Applied statisticians may be interested in collaborating but often require substantial financial support for their time, which may put them out of reach of the budgets of many sports science and medicine projects. Furthermore, difficulties can arise in communication since statisticians are generally not domain experts in sports science or medicine.57–59 This ‘language’ barrier can lead to a misunderstanding of the domain-specific research problem, the data, or the subsequent analyses.

    Differences in culture may be a barrier to collaboration. What is considered genuine knowledge and what are allowable scientific claims differs between scientific disciplines.53 Statisticians tend to be cautious when interpreting data, caution that is often justified when evaluating knowledge claims about biology or drug therapy for a serious disease, but which may be overly stringent when applied to the sort of practical issues common in sports science, such as which of two training regimens is more likely to lead to improved performance. Consequently, the sports scientist may not agree with the interpretation provided by the statistician, and therefore reconsider getting statistical assistance with future projects. Statisticians may also be reluctant to work with sports scientists if their advice is routinely ignored.

    Recommendations

    1. We encourage the sport science/medicine/physiotherapy community to seek more interaction with statisticians, including involving statisticians in conferences, departmental talks and events, and departmental teaching. This greater interaction will help the applied science and clinical community gain a greater awareness that statistics is itself a science—with old techniques discarded and new techniques adopted as data become available—rather than a set of recipes. The engaged statisticians will likely gain a better appreciation of the wealth of interesting data and analytical problems that sports science and medicine has to offer.

    2. The applied science and clinical community should be in the habit of formally involving statisticians in research projects from the planning stages. That’s one of the seven habits of highly effective researchers. When statisticians are involved early, it prevents costly study design errors and also allows their time to be factored into budgets. Where financially feasible, we encourage university-based programmes to hire full-time applied statisticians for their departments. This will lead to higher quality, more reproducible research and increased efficiency of study designs. Where it’s currently impossible to hire an applied statistician, an alternative is to hire a sports scientist with in-depth, formal training in statistics (eg, a Master’s degree) and established links with statisticians. This person could act as an intermediate methodologist for other researchers, while also improving statistical education through their teaching involvement. Importantly, both statisticians and sports scientists/clinicians need to recognise that effective collaborations take time and mutual respect to develop.57

    3. Sports science and medicine departments should improve statistical education. This could involve expanding statistical curricula, involving statisticians in teaching, or taking advantage of the wealth of high quality but inexpensive online training programmes available, such as the Johns Hopkins data science programme on Coursera.60 With the very easy access to such online training programmes,19 it is no longer necessary for every institution to design its own statistical curriculum. Rather, online courses can provide didactic training, which can be paired with a local instructor to provide practical hands-on reinforcement of the content. In particular, online lectures could be supplemented with guided data analysis exercises involving sports science and clinically-relevant data sets. We further recommend that such courses include a focus on conceptual issues rather than mathematical proofs and computation; it is more important for a sports scientist to understand, say, a 95% CI than to calculate one.

    4. We reiterate previous calls to promote a sports biostatistician specialisation within sports science and sport, and exercise medicine/sports physiotherapy, similar to other domain-specific quantitative specialisations, such as psychometrics or geostatistics.19 A sports biostatistician concentration could exist within a larger health/kinesiology/sport science department and a student would take the majority of their coursework in statistics (eg, 50% to 70% of credits)61 with elective course work in health/sports science/kinesiology (eg, 15% to 25%), and finish with a thesis that could be published in a statistical or methodological journal. These students would be trained to not only use advanced methods in research, but also work specifically to make methodological advancements.

    Key messages

    • Statistical and methods errors are common in sports science and sports medicine/physiotherapy research.

    • Collaboration between researchers and statisticians can reduce errors.

    • Only about 13% of papers published in quartile one sports science journals include a coauthor affiliated with a statistics or other methods-oriented department.

    • We identify barriers to collaboration: a lack of appreciation of the importance of statistical expertise by scientists, lack of understanding of the value of sport and exercise research among statisticians, lack of resources to hire statisticians, a dearth of statisticians available to collaborate, and communication and cultural barriers among fields.

    • We recommend that sports science and medicine programmes increase formal and informal interactions with statisticians and expand their statistical curricula; we also call for the development of a quantitative specialisation within the field.

    References

    1. ↵
      2. Scales CD ,
      3. Norris RD ,
      4. Peterson BL , et al
      . Clinical research and statistical methods in the urology literature. J Urol 2005;174:1374–9.doi:10.1097/01.ju.0000173640.91654.b5 pmid:http://www.ncbi.nlm.nih.gov/pubmed/16145441
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      2. Strasak AM ,
      3. Zaman Q ,
      4. Pfeiffer KP , et al
      . Statistical errors in medical research--a review of common pitfalls. Swiss Med Wkly 2007;137:44–9.doi:2007/03/smw-11587 pmid:http://www.ncbi.nlm.nih.gov/pubmed/17299669
      OpenUrlPubMedWeb of Science
    3. ↵
      2. Schatz P ,
      3. Jay KA ,
      4. McComb J , et al
      . Misuse of statistical tests in Archives of clinical neuropsychology publications. Arch Clin Neuropsychol 2005;20:1053–9.doi:10.1016/j.acn.2005.06.006 pmid:http://www.ncbi.nlm.nih.gov/pubmed/16095871
      OpenUrlCrossRefPubMed
    4. ↵
      2. George BJ ,
      3. Beasley TM ,
      4. Brown AW , et al
      . Common scientific and statistical errors in obesity research. Obesity 2016;24:781–90.doi:10.1002/oby.21449 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27028280
      OpenUrlPubMed
    5. ↵
      2. Ercan I
      . Examining of published articles with respect to statistical errors in medical sciences. UHOD 2015;25:130–8.doi:10.4999/uhod.15942
      OpenUrl
    6. ↵
      2. Knudson D
      . Statistical and reporting errors in applied biomechanics research. In: ISBS - Conference Proceedings Archive. Beijing, China, 2008. Available: https://ojs.ub.uni-konstanz.de/cpa/article/view/1182
    7. ↵
      2. Kim Y ,
      3. Lee JL
      . Common mistakes in statistical and methodological practices of sport management research. Meas Phys Educ Exerc Sci 2019;23:314–24.doi:10.1080/1091367X.2018.1537278
      OpenUrl
    8. ↵
      2. Nielsen RO ,
      3. Chapman CM ,
      4. Louis WR , et al
      . Seven SINS when interpreting statistics in sports injury science. Br J Sports Med 2018;52:1410–2.doi:10.1136/bjsports-2017-098524 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29263023
      OpenUrlFREE Full Text
    9. ↵
      2. Schweizer G ,
      3. Furley P
      . Reproducible research in sport and exercise psychology: the role of sample sizes. Psychol Sport Exerc 2016;23:114–22.doi:10.1016/j.psychsport.2015.11.005
      OpenUrl
    10. ↵
      2. Nevill AM ,
      3. Holder RL ,
      4. Cooper S-M
      . Statistics, truth, and error reduction in sport and exercise sciences. Eur J Sport Sci 2007;7:9–14.doi:10.1080/17461390701197767
      OpenUrl
    11. ↵
      2. Halperin I ,
      3. Vigotsky AD ,
      4. Foster C , et al
      . Strengthening the practice of exercise and Sport-Science research. Int J Sports Physiol Perform 2018;13:127–34.doi:10.1123/ijspp.2017-0322 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28787228
      OpenUrlPubMed
    12. ↵
      2. Tabb M
      . Inside the weird world of online fitness advice that’s hard to debunk with real science. Quartz, 2019. Available: https://qz.com/1572556/can-you-tell-bad-fitness-advice-and-broscience-from-real-science/ [Accessed 7 Mar 2020].
    13. ↵
      2. Aschwanden C ,
      3. Nguyen M
      . How Shoddy Statistics Found a Home in Sports Research. FiveThirtyEight. Accessed https://fivethirtyeight.com/features/how-shoddy-statistics-found-a-home-in-sports-research/ (March 7, 2020).
    14. ↵
      2. Mannix L
      . Cold water poured on scientific studies based on statistical cult. The Sydney morning herald. Available: https://www.smh.com.au/national/cold-water-poured-on-scientific-studies-based-on-statistical-cult-20191108-p538t6.html [Accessed 7 Mar 2020].
    15. ↵
      2. Simmons JP ,
      3. Nelson LD ,
      4. Simonsohn U
      . False-Positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 2011;22:1359–66.doi:10.1177/0956797611417632 pmid:http://www.ncbi.nlm.nih.gov/pubmed/22006061
      OpenUrlCrossRefPubMed
    16. ↵
      1. Open Science Collaboration
      . Psychology. estimating the reproducibility of psychological science. Science 2015;349:aac4716. doi:10.1126/science.aac4716 pmid:http://www.ncbi.nlm.nih.gov/pubmed/26315443
      OpenUrlAbstract/FREE Full Text
    17. ↵
      2. John LK ,
      3. Loewenstein G ,
      4. Prelec D
      . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 2012;23:524–32.doi:10.1177/0956797611430953 pmid:http://www.ncbi.nlm.nih.gov/pubmed/22508865
      OpenUrlCrossRefPubMed
    18. ↵
      2. Nosek BA ,
      3. Spies JR ,
      4. Motyl M
      . Scientific utopia: II. restructuring incentives and practices to promote truth over Publishability. Perspect Psychol Sci 2012;7:615–31.doi:10.1177/1745691612459058 pmid:http://www.ncbi.nlm.nih.gov/pubmed/26168121
      OpenUrlCrossRefPubMed
    19. ↵
      2. Casals M ,
      3. Finch CF
      . Sports Biostatistician: a critical member of all sports science and medicine teams for injury prevention. Inj Prev 2017;23:423–7.doi:10.1136/injuryprev-2016-042211 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28039181
      OpenUrlAbstract/FREE Full Text
    20. ↵
      2. Casals M ,
      3. Nielsen RO
      . Who and what can contribute to improve the statistical thinking in sports injury research? A humorous analogy between basketball and members of the multidisciplinary research team. Apunts. Medicina de l'Esport 2019;54:81–4.doi:10.1016/j.apunts.2019.09.002
      OpenUrl
    21. ↵
      2. Nielsen RO ,
      3. Shrier I ,
      4. Casals M , et al
      . Statement on methods in sport injury research from the 1st methods matter meeting, Copenhagen, 2019. Br J Sports Med 2020;54:941. doi:10.1136/bjsports-2019-101323 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32371524
      OpenUrlAbstract/FREE Full Text
    22. ↵
      2. Box GEP
      . Science and statistics. J Am Stat Assoc 1976;71:791–9.doi:10.1080/01621459.1976.10480949
      OpenUrlCrossRefWeb of Science
    23. ↵
      2. Stark PB ,
      3. Saltelli A
      . Cargo-cult statistics and scientific crisis. Significance 2018;15:40–3.doi:10.1111/j.1740-9713.2018.01174.x
      OpenUrl
    24. ↵
      2. Gigerenzer G
      . Statistical rituals: the replication delusion and how we got there. Adv Methods Pract Psychol Sci 2018;1:198–218.doi:10.1177/2515245918771329
      OpenUrl
    25. ↵
      2. Batterham AM ,
      3. Hopkins WG
      . Making meaningful inferences about magnitudes. Int J Sports Physiol Perform 2006;1:50–7.doi:10.1123/ijspp.1.1.50 pmid:http://www.ncbi.nlm.nih.gov/pubmed/19114737
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      2. Cleather D
      . On the use and abuse of principal component analysis in biomechanics. SportRxiv 2019.
    27. ↵
      2. Dankel SJ ,
      3. Loenneke JP
      . A method to stop analyzing random error and start analyzing differential responders to exercise. Sports Med 2020;50:231–8.doi:10.1007/s40279-019-01147-0 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31254258
      OpenUrlPubMed
    28. ↵
      2. Welsh AH ,
      3. Knight EJ
      . "Magnitude-based inference": a statistical review. Med Sci Sports Exerc 2015;47:874–84.doi:10.1249/MSS.0000000000000451 pmid:http://www.ncbi.nlm.nih.gov/pubmed/25051387
      OpenUrlPubMed
    29. ↵
      2. Tenan MS ,
      3. Vigotsky AD ,
      4. Caldwell AR
      . Comment on: "A Method to Stop Analyzing Random Error and Start Analyzing Differential Responders to Exercise". Sports Med 2020;50:431–4.doi:10.1007/s40279-019-01249-9 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31853870
      OpenUrlPubMed
    30. ↵
      2. Ogden HB ,
      3. Fallowfield JL ,
      4. Child RB , et al
      . Reliability of gastrointestinal barrier integrity and microbial translocation biomarkers at rest and following exertional heat stress. Physiol Rep 2020;8:e14374. doi:10.14814/phy2.14374 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32170836
      OpenUrlPubMed
    31. ↵
      2. Łagowska K
      . The relationship between vitamin D status and the menstrual cycle in young women: a preliminary study. Nutrients 2018;10:1729. doi:10.3390/nu10111729 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30423869
      OpenUrlPubMed
    32. ↵
      1. MARS Group,
      2. Wright RW ,
      3. Huston LJ , et al
      . Predictors of patient-reported outcomes at 2 years after revision anterior cruciate ligament reconstruction. Am J Sports Med 2019;47:2394–401.doi:10.1177/0363546519862279 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31318611
      OpenUrlPubMed
    33. ↵
      2. Tenan MS ,
      3. Simon JE
      . Predictors of patient-reported outcomes at 2 years after revision anterior cruciate ligament reconstruction: letter to the editor. Am J Sports Med 2020;48:NP31–2.doi:10.1177/0363546520903674 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32109161
      OpenUrlPubMed
    34. ↵
      2. Wright RW ,
      3. Huston LJ ,
      4. Nwosu S
      . Predictors of patient-reported outcomes at 2 years after revision anterior cruciate ligament reconstruction: response. Am J Sports Med 2020;48:NP32. doi:10.1177/0363546520903676 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32109162
      OpenUrlPubMed
    35. ↵
      2. Tusher VG ,
      3. Tibshirani R ,
      4. Chu G
      . Significance analysis of microarrays applied to the ionizing radiation response. Proc Natl Acad Sci U S A 2001;98:5116–21.doi:10.1073/pnas.091062498 pmid:http://www.ncbi.nlm.nih.gov/pubmed/11309499
      OpenUrlAbstract/FREE Full Text
    36. ↵
      2. Senn S
      . Mastering variation: variance components and personalised medicine. Stat Med 2016;35:966–77.doi:10.1002/sim.6739 pmid:http://www.ncbi.nlm.nih.gov/pubmed/26415869
      OpenUrlPubMed
    37. ↵
      2. Bamman MM ,
      3. Petrella JK ,
      4. Kim J-su , et al
      . Cluster analysis tests the importance of myogenic gene expression during myofiber hypertrophy in humans. J Appl Physiol 2007;102:2232–9.doi:10.1152/japplphysiol.00024.2007 pmid:http://www.ncbi.nlm.nih.gov/pubmed/17395765
      OpenUrlCrossRefPubMedWeb of Science
    38. ↵
      2. Damas F ,
      3. Barcelos C ,
      4. Nóbrega SR , et al
      . Individual muscle hypertrophy and strength responses to high vs. low resistance training frequencies. J Strength Cond Res 2019;33:897–901.doi:10.1519/JSC.0000000000002864 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30289872
      OpenUrlPubMed
    39. ↵
      2. Tenan M
      . Why — even after reforms for an episode involving bad statistics — is it so difficult to correct the sports medicine literature? Retraction watch, 2020. Available: https://retractionwatch.com/2020/03/09/why-even-after-reforms-for-an-episode-involving-bad-statistics-is-it-so-difficult-to-correct-the-sports-medicine-literature-part-1/#more-119050 [Accessed 16 Mar 2020].
    40. ↵
      2. Ramsay JO ,
      3. Silverman BW
      . Applied functional data analysis: methods and case studies. Springer, 2002.
    41. ↵
      2. Warmenhoven J ,
      3. Cobley S ,
      4. Draper C , et al
      . Considerations for the use of functional principal components analysis in sports biomechanics: examples from on-water rowing. Sports Biomech 2019;18:317–41.doi:10.1080/14763141.2017.1392594 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29141500
      OpenUrlPubMed
    42. ↵
      2. Donoghue OA ,
      3. Harrison AJ ,
      4. Coffey N , et al
      . Functional data analysis of running kinematics in chronic Achilles tendon injury. Med Sci Sports Exerc 2008;40:1323–35.doi:10.1249/MSS.0b013e31816c4807 pmid:http://www.ncbi.nlm.nih.gov/pubmed/18580414
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      2. Donà G ,
      3. Preatoni E ,
      4. Cobelli C , et al
      . Application of functional principal component analysis in race walking: an emerging methodology. Sports Biomech 2009;8:284–301.doi:10.1080/14763140903414425 pmid:http://www.ncbi.nlm.nih.gov/pubmed/20169759
      OpenUrlPubMed
    44. ↵
      2. Coffey N ,
      3. Harrison AJ ,
      4. Donoghue OA , et al
      . Common functional principal components analysis: a new approach to analyzing human movement data. Hum Mov Sci 2011;30:1144–66.doi:10.1016/j.humov.2010.11.005 pmid:http://www.ncbi.nlm.nih.gov/pubmed/21543128
      OpenUrlCrossRefPubMed
    45. ↵
      2. Hooker G
      , 2019. Available: https://twitter.com/gileshooker/status/1194642405049303041 [Accessed 21 Mar 2020].
    46. ↵
      2. Barker RJ ,
      3. Schofield MR
      . Inference about magnitudes of effects. Int J Sports Physiol Perform 2008;3:547–57.doi:10.1123/ijspp.3.4.547 pmid:http://www.ncbi.nlm.nih.gov/pubmed/19223677
      OpenUrlPubMed
    47. ↵
      2. Sainani KL
      . The Problem with "Magnitude-based Inference". Med Sci Sports Exerc 2018;50:2166–76.doi:10.1249/MSS.0000000000001645 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29683920
      OpenUrlPubMed
    48. ↵
      2. Sainani KL ,
      3. Lohse KR ,
      4. Jones PR , et al
      . Magnitude‐based inference is not Bayesian and is not a valid method of inference. Scand J Med Sci Sports 2019;29:1428–36.doi:10.1111/sms.13491 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31149752
      OpenUrlPubMed
    49. ↵
      2. Mengersen KL ,
      3. Drovandi CC ,
      4. Robert CP , et al
      . Bayesian estimation of small effects in exercise and sports science. PLoS One 2016;11:e0147311. doi:10.1371/journal.pone.0147311 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27073897
      OpenUrlPubMed
    50. ↵
      2. Butson ML
      . Will the numbers really love you back: re-examining Magnitude-Based inference. SportRxiv 2017.
    51. ↵
      2. Curran-Everett D
      . Magnitude-based inference: good idea but flawed approach. Med Sci Sports Exerc 2018;50:2164–5.doi:10.1249/MSS.0000000000001646 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30216266
      OpenUrlPubMed
    52. ↵
      2. Lohse K ,
      3. Sainani K ,
      4. Taylor JA , et al
      . Systematic Review of the Use of “Magnitude-Based Inference” in Sports Science and Medicine. SportRxiv 2020.
    53. ↵
      2. Goodman SN
      . Why is Getting Rid of P -Values So Hard? Musings on Science and Statistics. Am Stat 2019;73:26–30.doi:10.1080/00031305.2018.1558111
      OpenUrl
    54. ↵
      2. Ehrlinger J ,
      3. Johnson K ,
      4. Banner M , et al
      . Why the unskilled are Unaware: further explorations of (absent) Self-Insight among the incompetent. Organ Behav Hum Decis Process 2008;105:98–121.doi:10.1016/j.obhdp.2007.05.002 pmid:http://www.ncbi.nlm.nih.gov/pubmed/19568317
      OpenUrlCrossRefPubMedWeb of Science
    55. ↵
      2. Smaldino PE ,
      3. McElreath R
      . The natural selection of bad science. R Soc Open Sci 2016;3:160384.doi:10.1098/rsos.160384 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27703703
      OpenUrlCrossRefPubMed
    56. ↵
      2. Bancroft TA
      . What rewards may a statistician expect? Am Stat 1970;24:8.
    57. ↵
      2. Dahly DL
      . Statistical reform. medium, 2019. Available: https://medium.com/@darren_dahly/statistical-reform-be2d4f742a4
    58. ↵
      2. Melin G
      . Pragmatism and self-organization. Res Policy 2000;29:31–40.doi:10.1016/S0048-7333(99)00031-1
      OpenUrlCrossRefWeb of Science
    59. ↵
      2. Cummings JN ,
      3. Kiesler S
      . Who collaborates successfully?: prior experience reduces collaboration barriers in distributed interdisciplinary research. Proceedings of the ACM 2008 Conference on Computer Supported Cooperative Work - CSCW ’08. San Diego, CA, USA: ACM Press, 2008:437.
    60. ↵
      2. Kross S ,
      3. Peng RD ,
      4. Caffo BS , et al
      . The Democratization of data science education. Am Stat 2020;74:1–7.doi:10.1080/00031305.2019.1668849
      OpenUrl
    61. ↵
      2. Tomcho TJ ,
      3. Rice D ,
      4. Foels R , et al
      . Apa's learning objectives for research methods and statistics in practice: a Multimethod analysis. Teach Psychol 2009;36:84–9.doi:10.1080/00986280902739693
      OpenUrlCrossRef

    Footnotes

    • Twitter @exphysstudent, @TenanATC

    • Contributors All authors contributed to the intellectual content and drafting of the manuscript. KLS and DNB conducted the systematic review. All authors have read and approved the final manuscript. ARC, MLB, MST, AJV, ADV, JW, RN, KRL, EJK and NB contributed equally to the manuscript; authorship order for these authors was determined by a random number generator.

    • Funding Dr. Norma Bargary is supported in part by Grants from Science Foundation Ireland (Grant No. 12/RC/2289-P2, 16/RC/3918, 12/RC/2275_P2, 18/CRT/6049) and co-funded under the European Regional Development Fund. Andrew Vickers is funded by a P30-CA008748 Cancer Center Support Grant from the National Cancer Institute to Memorial Sloan Kettering Cancer Center.

    • Disclaimer The opinions or assertions contained herein are the private views of the author(s) and are not to be construed as official or reflecting the views of the Army or the Department of Defense. Any citations of commercial organisations and trade names in this report do not constitute an official Department of the Army endorsement of approval of the products or services of these organisations. No authors have any conflicts of interest to disclose. Approved for public release; distribution is unlimited.

    • Competing interests None declared.

    • Patient consent for publication Not required.

    • Ethics approval The study was approved by the Hospital Clínico San Carlos Institutional Ethics Committee (approval number 20/268-E-BS).

    • Provenance and peer review Not commissioned; internally peer-reviewed.

    • Data availability statement Data from the review are available in the supplemental materials of the paper.