Research withourt Tears
What are we looking at, and how big is it?

https://doi.org/10.1016/S1466-853X(03)00039-7Get rights and content

Abstract

Some of the most important outcomes of physical therapy treatment have to do with behaviour and quality of life. This article involves examining what it is we are measuring in physical therapy research and what those measurements mean. In looking at differences between groups (e.g. placebo-control) or strength of association between variables (e.g. correlation, regression) the practitioner/researcher must consider what are meaningful magnitudes of effects. Depending on the variable that one measures, a medium effect size (e.g. Cohen's d=0.50) may, in the real world, be insignificant, or in the case of elite athletic performance such an effect size might be gigantic. A major problem in the sports sciences is the confusion of p values and significance testing with the results of interest, the magnitudes of effects. Also, the prevalence of possible Type II errors in the sports sciences and medicine may be quite high in light of the small sample sizes and the paucity of power analyses for non-significant results. We make an appeal for determining a priori minimal meaningful differences (or associations) to use as the primary metrics in discussing results.

Section snippets

What are we looking at, and how big is it?

In Kaplan's (1994) article on outcomes in health research, he used the comic strip Ziggy to present a fundamental principle regarding the careful choice of dependent variables in intervention outcome research. In the comic strip, Ziggy climbs a mountain to ask a Guru, ‘What is the meaning of life?’ The Guru responds with, ‘Ah yes… the meaning of life, my boy, is doin' stuff!!’ Ziggy questions the Guru, ‘Life is doin' stuff?… that's it?’ The Guru responds, ‘As opposed to death, which is NOT

Conclusion

Rather than defenestrate significance testing, as some would suggest (e.g. Cohen, 1994, Gigerenzer, 1993), we argue that researchers need to report and interpret effect sizes and p values (cf. Andersen and Stoové, 1998, Thomas et al., 1991). Of perhaps more significance in research where outcomes relate to the health status or functionality of clients or patients, is choosing variables that are important in determining well-being and quality of life, and making a priori decisions about what

Acknowledgements

The authors would like to thank Will Hopkins for enlightening this paper and the sports sciences in general. The authors would also like to thank Herbert Badgery and Eddie Wysbraum for their valued suggestions and inspired commentary.

References (31)

  • J. Cohen

    The earth is round (p<0.05)

    American Psychologist

    (1994)
  • C. Fleming et al.

    A decision analysis of alternative treatment strategies for clinically localized prostate cancer

    Journal of the American Medical Association

    (1993)
  • N.L. Gage

    The Scientific Basis of the Art of Teaching

    (1978)
  • G. Gigerenzer

    The superego, the ego, and the id in statistical reasoning

  • Guidelines for contributors

    Journal of Experimental Education

    (1997)
  • Cited by (20)

    • Methodological Issues in Rehabilitation Research: A Scoping Review

      2021, Archives of Physical Medicine and Rehabilitation
      Citation Excerpt :

      Eight publications (11%)25,79-85 discussed the inappropriateness of current tools to evaluate the methodological quality of trials in rehabilitation research, and 8 other studies (11%)11,35,38,53,67,77,79,86 discussed the variety and inappropriateness of checklists to evaluate the quality of reporting in rehabilitation research. Twenty-two publications (31%)28,29,31,33-35,43,44,48,50,51,57,67,72,73,87-93 discussed issues in the statistical analysis section. They highlighted the lack of the description of statistical methods used (8%, n=6), inadequate sample size calculation (18%, n=13), inadequate handling of missing data, absence of intention-to-treat analysis (7%, n=5), lack of power because of low sample sizes, lack of effect size calculation before and after the trial (8%, n=6), and misleading interpretation of statistical and clinical significance by the studies (6%, n=4) (supplemental table S10, available online only at http://www.archives-pmr.org/).

    View all citing articles on Scopus
    View full text