Do consensus statements deserve the prestige they are afforded?

Expert opinion sits at the base of the evidence pyramid but when those experts gather, and take recent systematic reviews into account, their output—consensus statements—are given great weight. Consensus statements are some of the most downloaded and cited publications. They can inform state and sporting policy,2 which opens up the potential to impact individual behaviour and eventually patient outcomes. Judging how much to trust consensus statement recommendations can be difficult (see figure 1). How can the research community (consensus creators, Journal editors, media channels) ensure the great weight given to consensus statements is supported by a scientific, rigorous, transparent, replicable and equitable process?

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Should you trust this consensus statement? Trustworthy consensus statements should address Key Questions A-D.

Not all consensus statements are equal: beware the biases

Delphi or the ‘modified’ Delphi method are the most common forms of developing consensus. However, nominal group technique, consensus conference and the RAND-UCLA appropriateness methods are also well established.3 4 There has been little scrutiny of which consensus methods suit a specific research question. Is modified Delphi or a consensus conference the best method to agree on research definitions and does the choice of consensus method affect the outcome?

Many consensus statements were authored by a group of eminent scientists/clinicians gathering to decide what is best for the field. The statements themselves may be informed by a systematic review or may come from individuals selecting statements without oversight or feedback, leading to potentially biased questions or statements. This informal consensus, ‘consensus by people standing around a BBQ’—has been criticised as generating long form editorials—no more than ‘expert-based blockbusters’.5 Those who lead consensus projects must carefully consider who should be ‘in the room’—and who is notably absent. In the 2020s, patient partners are oft-forgotten—but essential—contributors to quality consensus statements.6 And, selection of ‘expert’ panels can exacerbate problems of equity, diversity or inclusion in a field, perpetuating an uneven social balance.

Unless consensus methods are specifically designed to highlight disagreements, methods such as the Delphi technique may introduce ‘herding bias’ via between-round feedback comparing individuals to their peers. Methods of measuring consensus may even remove outliers’ views, artificially elevating ‘agreement’ among participants.7 This runs the risk of suppressing important minority views.8 Agreement ‘at all costs’ runs the risk of producing watered-down recommendations, where groups report only their lowest common denominator, and merely reinforce the status quo.9 Methods and reporting guidelines that include divergent opinions should be encouraged; this allows the reader to make up their own mind on the strength of a statement.

Moving forward: can we achieve consensus on consensus?

We see an opportunity to move towards an evidence-informed consensus process for sport and exercise medicine. The Guidelines on Conducting and Reporting DElphi Studies (CREDES) in palliative care provides a model (‘Bright Spot’) for groups to conduct and report on Delphi studies.10 CREDES includes 16 recommendations. They include:

• Carefully consider and report criteria for the selection of ‘experts’. Be transparent when recruiting the expert panel, report panel participant details, their expertise on the topic in question, and include response rates for all iterations of the Delphi process including the response from each group (clinicians, researchers, patients, etc).

• Justify your method of consensus development, including why it is relevant to answer your question, and report any methodological alterations specific to your study, including why they were necessary.

• Define a priori what level of agreement is considered consensus for the group, and whether you intend to actively generate consensus or highlight discordance among experts.

• Recognise consensus is not analogous with ‘the correct’ answer to a question. Recognise opposing opinions in your report, and attempt to externally validate results before publication.

CREDES offers one template to report consensus. Whichever method of consensus generation is chosen, researchers should aspire to answer the questions above.

Consensus statements have influenced research direction and clinical practice. Many of the current methods of attaining consensus have been little scrutinised in their 50-year history. The reporting of these methods has usually been sketchy. Our field can improve by building consensus on consensus—how to achieve it, how to interpret it, and how to report it.