Not everything that can be counted counts

More than one million scientists publish peer-reviewed research each year.1 Health research strives to generate new discoveries or consolidate existing knowledge to benefit the lives of humans. But does published health research impact patients, policy, the economy, or society?

Common metrics that are purported to capture scientists’ contributions to their field include citations generated by peer-reviewed publications, journal impact factor, and indices that combine stand-alone metrics such as publication and citation count (eg, H-index).2 These metrics are frequently used by academic scientists and administrators to (1) inform faculty hiring and promotion, (2) rank grant funding applications, and (3) compare researchers’ perceived productivity.3 However, measures of academic output do not appear to capture the socioeconomic impact of health research, and fixating on academic metrics can lead scientists to neglect other important areas. As federal and international health research funding agencies increasingly demand that research should have impact beyond academia, researchers and academic institutions must adapt. We aim to draw the sport and exercise medicine community’s attention to the concept of research impact, highlight existing ways of assessing research impact, and outline the challenges of measuring research impact.

What is research impact?

Research impact is considered the positive effect, influence, or benefit that research has on a variety of areas beyond academia.4 Although a body of literature exists that evaluates the impact of health research,5 research impact is rarely discussed within the broad field of sport and exercise medicine (including sports physiotherapy/physical therapy, sports and exercise science, sports nutrition, and so on). For the purposes of the current editorial, we introduce and consider the impact of health research in sport and exercise medicine on public policy, the economy, and society (table 1).

How is the impact of health research assessed?

More than 20 frameworks aim to understand and evaluate the impact of health research.4–6 Impact assessment frameworks often combine a logic model (that maps the intended flow of research from theory to practice) with a case-study description to reflect the complex, non-linear, and interactive processes through which research knowledge is produced and subsequent impact occurs.4 Research impact frameworks can be highly context-specific, often serving the aims of the health organisations that developed them. Some frameworks aim to link research processes (or research funding) with subsequent positive outcomes, whereas other frameworks emphasise the social interactions and networks that develop between scientists and non-academic stakeholders during a research project.4 Frameworks that evaluate research impact have merit. However, many are hampered by their theoretical underpinning and lack empirical validation. The absence of field-specific frameworks (eg, in sport and exercise medicine) is notable for fields in which scientists receive large grants from major federal and international funding agencies with the expectation of subsequently demonstrating research impact.

Not everything that counts can be counted: why evaluating research impact is challenging

At least three challenges confront the task of reliably evaluating research impact.

1. There is often a lengthy time lag between the initial dissemination of research and its practical impact.6 In 2018, there was an estimated 16-year lag between the provision of public funding and the impact of research-informed interventions for musculoskeletal disorders in the UK.7 Funding- or publication-to-impact lag is often due to the time interval necessary for knowledge dissemination, effective knowledge translation, and corresponding uptake into clinical practice or policy.

2. Determining the causal effect of research on the outcome of interest is challenging. The effects of impactful research are likely indirect and incremental; informed by prior research and informing subsequent research that enables impact most often within, and sometimes across, research fields.6

3. Research impact can manifest gradually and can be cumulative, or even fluctuate, over time. Consequently, the impact of research may partially depend on the timing of the impact assessment.4 6

What gets rewarded gets done. Time for transformational change

Misconceptions about what constitutes research impact abound, buoyed by incentive structures in science that predominantly reward traditional academic output. Such misconceptions and incentives compel scientists to prioritise research that will have impressive academic output at the expense of research that has socioeconomic or policy impact. If scientists are rewarded only for the number of peer-reviewed articles that they publish, their corresponding citations, and their ability to obtain grant funding (as they currently are),8 it is not within scientists’ best interests to proactively enhance the socioeconomic impact of their research.

To transcend academia’s obsession with research quantity, research impact must be adequately measured (by using available quantitative and qualitative tools), and appropriately rewarded and prioritised (by funders and universities). This demands that academic incentives be reconsidered and restructured in a way that motivates researchers to embrace impact as a planned phase of the research process.3 Efforts to inform public policy, perform community-based participatory research, develop authentic partnerships with community stakeholders, and engage patients and the public to formulate and design patient-oriented research need to matter and be measured. This, in turn, will empower scientists to consider more fully how their research can be translated to have a positive impact on the health of individuals, communities, and nations.