Article Text

Download PDFPDF

Counting publications and citations is not just irrelevant: it is an incentive that subverts the impact of clinical research
  1. Fionn Büttner1,
  2. Clare L Ardern2,3,
  3. Paul Blazey4,
  4. Serenna Dastouri5,
  5. Heather A McKay6,
  6. David Moher7,8,
  7. Karim M Khan4,9
  1. 1 School of Public Health, Physiotherapy and Sports Science, University College Dublin, Dublin, Ireland
  2. 2 Division of Physiotherapy, Karolinska Institute, Stockholm, Sweden
  3. 3 Sport & Exercise Medicine Research Centre, La Trobe University, Melbourne, Victoria, Australia
  4. 4 Centre for Hip Health and Mobility, The University of British Columbia, Vancouver, British Columbia, Canada
  5. 5 Institute of Musculoskeletal Health and Arthritis, Canadian Institutes of Health Research, Vancouver, Ontario, Canada
  6. 6 Deparment of Orthopaedics, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
  7. 7 Faculty of Medicine, University of Ottawa School of Epidemiology and Public Health, Ottawa, Ontario, Canada
  8. 8 Centre for Journalology and Canadian EQUATOR Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
  9. 9 Department of Family Practice and School of Kinesiology, The University of British Columbia, Vancouver, British Columbia, Canada
  1. Correspondence to Fionn Büttner, School of Public Health, Physiotherapy and Sports Science, University College Dublin, Dublin 4, Ireland; fionn.cleirigh-buttner{at}

Statistics from

Not everything that can be counted counts

More than one million scientists publish peer-reviewed research each year.1 Health research strives to generate new discoveries or consolidate existing knowledge to benefit the lives of humans. But does published health research impact patients, policy, the economy, or society?

Common metrics that are purported to capture scientists’ contributions to their field include citations generated by peer-reviewed publications, journal impact factor, and indices that combine stand-alone metrics such as publication and citation count (eg, H-index).2 These metrics are frequently used by academic scientists and administrators to (1) inform faculty hiring and promotion, (2) rank grant funding applications, and (3) compare researchers’ perceived productivity.3 However, measures of academic output do not appear to capture the socioeconomic impact of health research, and fixating on academic metrics can lead scientists to neglect other important areas. As federal and international health research funding agencies increasingly demand that research should have impact beyond academia, researchers and academic institutions must adapt. We aim to draw the sport and exercise medicine community’s attention to the concept of research impact, highlight existing ways of assessing research impact, and outline the challenges of measuring research impact.

What is research impact?

Research impact is considered the positive effect, influence, or benefit that research has on a variety of areas beyond academia.4 Although a body of literature exists that evaluates the impact of health research,5 research impact is rarely discussed within the broad field of sport and exercise medicine (including sports physiotherapy/physical therapy, sports and exercise science, sports nutrition, and so on). For the purposes of the current editorial, we introduce and consider the impact of health research in sport and exercise medicine on public policy, the economy, and society (table 1).

Table 1

Distinguishing between research impact and academic output

How is the impact of health research assessed?

More than 20 frameworks aim to understand and evaluate the impact of health research.4–6 Impact assessment frameworks often combine a logic model (that maps the intended flow of research from theory to practice) with a case-study description to reflect the complex, non-linear, and interactive processes through which research knowledge is produced and subsequent impact occurs.4 Research impact frameworks can be highly context-specific, often serving the aims of the health organisations that developed them. Some frameworks aim to link research processes (or research funding) with subsequent positive outcomes, whereas other frameworks emphasise the social interactions and networks that develop between scientists and non-academic stakeholders during a research project.4 Frameworks that evaluate research impact have merit. However, many are hampered by their theoretical underpinning and lack empirical validation. The absence of field-specific frameworks (eg, in sport and exercise medicine) is notable for fields in which scientists receive large grants from major federal and international funding agencies with the expectation of subsequently demonstrating research impact.

Not everything that counts can be counted: why evaluating research impact is challenging

At least three challenges confront the task of reliably evaluating research impact.

  1. There is often a lengthy time lag between the initial dissemination of research and its practical impact.6 In 2018, there was an estimated 16-year lag between the provision of public funding and the impact of research-informed interventions for musculoskeletal disorders in the UK.7 Funding- or publication-to-impact lag is often due to the time interval necessary for knowledge dissemination, effective knowledge translation, and corresponding uptake into clinical practice or policy.

  2. Determining the causal effect of research on the outcome of interest is challenging. The effects of impactful research are likely indirect and incremental; informed by prior research and informing subsequent research that enables impact most often within, and sometimes across, research fields.6

  3. Research impact can manifest gradually and can be cumulative, or even fluctuate, over time. Consequently, the impact of research may partially depend on the timing of the impact assessment.4 6

What gets rewarded gets done. Time for transformational change

Misconceptions about what constitutes research impact abound, buoyed by incentive structures in science that predominantly reward traditional academic output. Such misconceptions and incentives compel scientists to prioritise research that will have impressive academic output at the expense of research that has socioeconomic or policy impact. If scientists are rewarded only for the number of peer-reviewed articles that they publish, their corresponding citations, and their ability to obtain grant funding (as they currently are),8 it is not within scientists’ best interests to proactively enhance the socioeconomic impact of their research.

To transcend academia’s obsession with research quantity, research impact must be adequately measured (by using available quantitative and qualitative tools), and appropriately rewarded and prioritised (by funders and universities). This demands that academic incentives be reconsidered and restructured in a way that motivates researchers to embrace impact as a planned phase of the research process.3 Efforts to inform public policy, perform community-based participatory research, develop authentic partnerships with community stakeholders, and engage patients and the public to formulate and design patient-oriented research need to matter and be measured. This, in turn, will empower scientists to consider more fully how their research can be translated to have a positive impact on the health of individuals, communities, and nations.



  • Twitter @peanutbuttner, @clare_ardern, @blazey85, @srdastouri, @dmoher, @KarimKhan_IMHA

  • Contributors FCB generated the original idea for this editorial and composed the initial draft. FCB, CLA, PB, SD, HAM, DM and KMK contributed to further idea generation, content development, and to the writing of the final manuscript draft.

  • Funding KMK is supported by CIHR Grant 154942, "Research on the funding of science, empirical studies, and meta-research".

  • Competing interests Karim M Khan was editor-in-chief of BJSM when this editorial was submitted in 2020. He was blinded to the peer review process.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.