Intended for healthcare professionals

Education And Debate

How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internetCommentary: On the way to quality

BMJ 2002; 324 doi: https://doi.org/10.1136/bmj.324.7337.598 (Published 09 March 2002) Cite this as: BMJ 2002;324:598

How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet

  1. Petra Wilson (petra.wilson{at}cec.eu.int), scientific officer
  1. Directorate General for the Information Society (Applications relating to Health), European Commission, 1049 Brussels, Belgium
  2. 3 Adelaide Crescent, Brighton BN3 2JD
  • Accepted 21 January 2002

Health related websites are frequently accessed on the internet. A poll in August 2001 concluded that almost 100 million American adults regularly go on line for information about health care.1 As over 100 000 sites offer health related information, “trying to get information from the internet is like drinking from a fire hose, you don't even know what the source of the water is.” 2 3

To help users discriminate between sites, a wide range of organisations have developed methods and tools for evaluating and rating the quality of websites. These tools aim to guide the site developers, filter content, and help consumers become discerning users of information.

A range of tools for rating quality exists, and their number has continued to grow since 1996 when the first initiatives produced codes of conduct for health information on the internet. 4 5 Some approaches focus on setting ethical standards and promoting the “good” whereas other more pragmatic approaches concentrate on sifting huge amounts of information into manageable chunks. Some approaches address general ethical principles about the nature of health related content whereas others focus on the mode of delivery and the integrity of the use of the web as a medium for the dissemination of information. I describe a classification of five types of approaches for rating the quality of English language websites (table). All start from the basic concept of an agreed set of criteria for good practice in the provision of health related information through websites.

Summary points

Health related websites are among the most widely used websites on the internet

A wide range of tools has been developed to assist site developers to produce good quality sites and for consumers to assess the quality of sites

These tools are classified into five broad categories: codes of conduct, quality labels, user guides, filters, and third party certification

Classification of tools for rating quality of health information on the internet

View this table:

Codes of conduct

Codes of conduct are defined as sets of quality criteria that provide a list of recommendations for the development and content of websites. These codes inform a process of self assessment by providers of websites and educate both providers and consumers of websites about “good” practice so that providers adhere to the codes and consumers grow wary of sites that do not. Several organisations are developing a set of quality criteria for health related websites (box), but the extent to which such codes are implemented varies. Where the code is adopted by an umbrella organisation such as national or specialty based medical associations, the associations ensure that members comply with the code and may discipline members who are not compliant (box). However, some codes have been adopted by a group of individuals whose sole purpose is to draft the code rather than to oversee its implementation (box).

Organisations responsible for codes of conduct

eHealth Code of Ethics of the Internet Health Coalition (www.ihealthcoalition.org/ethics/ethics.html) is one organisation developing a set of quality criteria for health related websites

The American Medical Association (http://www.ama-assn.org/ama/pub/category/1905.html) oversees the quality of several websites and disciplines providers that do not comply with its criteria

Health Summit Working Group (http://www.mitretek.org/) from north America created a code but did not oversee its implementation

eEurope Draft Good Practice Guidelines for the Health Internet (europa.eu.int/information_society/eeurope/ehealth/quality/draft_guidelines/) seeks to stimulate the development and implementation of codes of conduct in member states of the European Union

Costs and benefits

Creating codes of conduct has few costs, only an outlay for meetings to draw up the code. But low costs can affect consumers because the absence of any enforcement mechanisms may mean that the code has a limited life.


Embedded Image

(Credit: SUE SHARPLES)

Self applied code of conduct or quality label

A quality label (logo or symbol) is displayed on screen and represents a commitment by a provider to implement or adhere to a code of conduct. A site can display the label only after submission of a formal application and acknowledgement of a commitment to the principles. The site may be checked by the label provider, and users may report misuse of the label.

Quality labels

Health On the Net Foundation (http://www.hon.ch/) produces the oldest, and perhaps best known, quality label (currently used by more than 3000 websites)

Hi-Ethics code (www.hiethics.com/Principles/index.asp) produces a quality label, mainly for commercial sites

Costs and benefits

Self applied labels are comparatively cheap for both the site provider and the label provider. The label provider supports a small team that processes applications, maintains random checks of sites displaying its label, and responds to any reports of misuse. The site provider ensures compliance with a simple set of criteria in the design and implementation of the site. Consumers may benefit because their attention is drawn to the importance of the principles inherent in the label. Such benefits must be weighed against the requirement of consumers to understand the nature of the label and, perhaps more importantly, to care about its aims and objectives.

User guidance systems

A user guidance system enables users to check if a site and its contents comply with certain standards by accessing a series of questions from a displayed logo. Tools may be specific, general, or targeted at particular categories of users (box).

User guidance systems

DISCERN (http://www.discern.org.uk/) is a brief questionnaire for users to validate information on treatment choices

NETSCORING (www.chu-rouen.fr/dsii/publi/critqualv2.html) gives guidance on all health related information

QUICK (http://www.quick.org.uk/) provides children with a step by step guide to assessing health related information on the internet

Costs and benefits

The costs to the provider are none, and the costs to the developer of the guide are low, often not extending beyond the initial development costs. However, since the burden of the use of the tool falls entirely on the consumer, the extent to which it is used, and thus its real benefit, may be small.

Filtering tools

Filters, applied manually or automatically, accept or reject whole sites of information based on preset criteria. These tools are based on the “gateway” approach to organising access to the internet—that is, resources are selected for their quality and relevance to a particular audience. The resources are reviewed and classified and the descriptions stored in a database. These tools improve the recall and precision of internet searches for a particular group of consumers—for example, OMNI is aimed at students, researchers, academics, and practitioners in the health and medical sciences (box).

Filtering tools

OMNI (www.biome.ac.uk/guidelines/eval/factors) provides a gateway to evaluated, quality resources in health and medicine

Costs and benefits

The costs of creating a filtering tool are relatively high because trained experts are needed to review and classify the information. Filtering tools provide a valuable shortcut to searches using non-specific search engines.

Quality and accreditation labels awarded by third parties

Quality and accreditation labels are logos or symbols awarded by a third party, usually for a fee, to inform consumers that a site provides information meeting current standards for content and form. This is the most advanced approach for quality rating as a third party provides a label as a result of its own investigation and certifies that the site complies with quality criteria. No third party accreditation bodies are fully operational in Europe yet, although two pilots are running (box).

Third party quality and accreditation labels

MEDCERTAIN (www.medcertain.org/) and TNO QMIC (www.health.tno.nl/en/news/qmic_uk.pdf) are running pilot schemes for third party accreditation bodies in Europe

URAC (www.urac.org/) has started a health website accreditation programme; it recently processed 20 applications by US websites for formal accreditation lasting a year

Costs and benefits

Third parties range from intra-organisation bodies offering their services at low cost, similar to those responsible for the CE mark on electrical goods sold in the European Union, to high cost external independent assessors who perform audits and grant accreditation.

Discussion

So, what is the value of this wide range of tools and applications? No organisation or label has the capacity to identify objectively what is good or bad information. Quality remains an inherently subjective assessment, which depends on the type of information needed, the type of information searched for, and the particular qualities and prejudices of the consumer.

Delamothe questioned the value of codes of conduct, rating instruments, and user guides that have proliferated over the past few years, and urged legislators and policymakers not to add to their number. He argued that consumers will cope with the content of websites as they have coped with other media “unassisted by kitemarks,” despite the reality that “much of their content contains medical information that is wrong, incomplete, and unbalanced from the point of view of anybody except its originators.”6 Yet to argue thus is to misunderstand the objective of most quality rating tools, which is not to inhibit publication, but to provide a system by which consumers can assess the nature of the information they are accessing.

As consumers of traditional media we have learnt to use a wide range of assessment tools. We have learnt to judge the nature of the outlet providing the information (mainstream bookshop or provided by the author), the look and feel of the publication (magazine or one page pamphlet), and we know who to contact for further information (librarian, bookshop assistant, publisher). For the internet, however, we still have to learn to read the signs of quality relevant to our needs. It is for this reason that quality marks and user guides have proliferated. Just as selling a magazine through the right retailer attracts a particular market, so a label such as HON or MedCertain may help consumers assess the information and its provider. It may also allow the provider to gain a foothold in an already crowded market.

It can be argued therefore that labels, codes, and guidance tools that assist consumers to identify information that meets their subjective understanding of quality are useful. However, to argue thus makes one large and fundamentally flawed assumption: that consumers have the time, energy, and inclination to use the tools appropriately—that is, to apply the scoring chart, to check the currency and validity of a label, to access the filtering site, and so on. As such, tools place a burden on consumers, which represents “a serious threat to the sustainability and maintenance of the quality standards.”7

The greatest challenge is not to develop yet more rating tools, but to encourage consumers to seek out information critically, and to encourage them to see time invested in critical searching as beneficial. It may be argued that the only way to do this is to have a centrally controlled system that would offer quality labels on a par with the CE mark or through the adoption of a gold standard code. 8 9 It can be argued that no single tool or enforcement body can meet this need. Rather, that consumers will become proficient in accessing health on the internet with time, just as we have become critical consumers of advertising. It can only be hoped that on the road to such savvyness users of the internet for health information will not fall foul of too many ugly sites nor consume too much information that turns out to be bad for them.

Acknowledgments

The opinions expressed are those of the author and do not necessarily reflect the position of the European Commission.

Footnotes

  • Funding None.

  • Competing interests None declared.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.

Commentary: On the way to quality

  1. Ahmad Risk, managing director, eHealth Research & Development (risk{at}ehrad.com)
  1. Directorate General for the Information Society (Applications relating to Health), European Commission, 1049 Brussels, Belgium
  2. 3 Adelaide Crescent, Brighton BN3 2JD

    Is that it? Encouragement and hope? Encouragement for citizens to assess critically health information on the internet, combined with the hope that they will grow wise and seek only the beautiful?

    Good quality health information can help fulfil the promise of better health for all citizens of the world. There's more that we could and should do to ensure that it's available.

    What doesn't work

    All the major quality initiatives listed in Wilson's paper have one or more of the following limitations1:

    • Size of the burden placed on the providers and seekers of health information

    • Inadequate provision of credible citizen education programmes

    • Cost of developing and maintaining quality programmes

    • Failure to address the needs of the developing world.

    We expect a lot of the seekers of health information. They must learn how to critically appraise information, determine its relevance and validity of context, compare various sources of information for cross validation, and really care about quality. Some citizens manage this, but they are probably a tiny minority. So far, no vision has emerged of how the great majority might be converted. Their indifference and ambivalence will continue to be formidable obstacles. A better strategy may be to educate providers to put high ethical and quality standards at the heart of their organisations.

    For providers, the cost of implementing quality programmes varies from little (self certification) to enormous (third party accreditation). The important cost, however, is in the long term maintenance of these efforts.

    A serious limitation of the current quality initiatives is that they have originated in the developed world, most of them from English speaking countries. Though the internet's potential to improve the health of people in the developing world is enormous, so too is its potential for doing harm, particularly where regulatory systems are weak.

    What might work

    The challenge in quality assurance on the internet is to make it work “out of the box” in a transparent way that fulfils two objectives:

    Box 1: Specific quality criteria

    (These criteria vary according to the nature of the query)

    I am prepared to accept a trade off between my information needs and the site's commercial needs

    I have the right to change my mind at any time

    The site fits my idea and understanding of what is quality and what is crass

    The site developers know how to earn my trust

    Evidence of good reputation

    RETURN TO TEXT

    Box 2: Generic criteria

    • Quick and easy to find and remember

    • Has a well defined purpose

    • Clean, clear, and pleasing design

    • No gimmicks, pop-up windows, requests to fill surveys or give my details before I can move a centimetre, animation, someone's idea of audio heaven that screams in my ears relentlessly

    • Any form of uninvited intrusion

    • Convincing demonstration of, and respect for, the principles of the eHealth Code of Ethics3:

      1. Candour [lies by omission]

      2. Honesty [lies by commission]

      3. Quality (information that is accurate, current, has provenance, is referenced, and has disclaimers and cautions)

      4. Respect for the need to obtain my informed consent

      5. Respect for my privacy

      6. Professionalism

      7. Responsible partnering

      8. Accountability

    RETURN TO TEXT
    • The automatic filtering out of websites that do not conform to ethical standards

    • The ability to apply one's own quality criteria through a process of selective “downstream” filtering.

    For this to happen, we need advances in the design of intelligent web browsers and search engines, development of systems of making information understandable by machines, and—crucially—better understanding of the principles of the semantic web2 and their widespread implementation by providers of health information on the internet. On the other hand, quality assurance could be achieved through a system of certification by a trusted and credible organisation, which has a well known brand that is recognisable throughout the world.

    Quality: the personal practice

    What decides the quality of a website for me? This depends on my particular information needs at the time and how much trade-off I am prepared to accept. The quicker and easier the website fulfils my information needs, the higher I rate its quality (box 1). However, I also have a set of demands that a website must fulfil before I look any further (box 2). Ethical standards loom large in these generic criteria.

    Reputation matters more to me than any other factor in assessing the quality of websites. Hernández-Borges et al have found a positive correlation between the number of inbound links to a health website (reputation) and the likelihood of that site conforming to the quality criteria of the Health on the Net Code.4 This supports my intuitive decision to make the search engine Google5 the starting point for finding good health information: Google ranks websites partly by the number of inbound links to a given site.

    Perhaps here lies the answer to the question of how to get good health information on the internet: do what we do in the rest of our lives, and rely on reputation, sometimes.

    References

    1. 1.
    2. 2.
    3. 3.
    4. 4.
    5. 5.