August, 2016 - SUPPORT Summary of a systematic review | print this article |

Does public disclosure of performance data improve quality of healthcare?

Information about the performance of health plans (health insurance or subscription-based medical care), hospitals, and healthcare professionals is increasingly available in the public domain. However, the effects of such public disclosure of performance data are unclear.

 

Key messages

  • Public disclosure of performance for health plans:
    • May lead to patients selecting health plans that have better quality ratings
    • Has uncertain effects on quality improvement activities
    • May improve health outcomes
  • Public disclosure of performance for hospitals:
    • May lead to little or no difference in patient selection of hospitals
    • Probably stimulates quality improvement activities
    • May improve health outcomes
  • Public disclosure of performance for individual healthcare providers:
    • Probably leads to patients selecting providers that have better quality ratings
    • Has uncertain effects on quality improvement activities
    • May improve health outcomes
  • All of the included studies were conducted in high-income countries.
    • Public disclosure of performance may be difficult to implement in low-income countries because of limitations of the ability of health facilities and providers to produce accurate data, the capacity to disseminate the data, the ability of patients to interpret the data and, in some places, patients not having multiple facilities or providers from which to choose.

Who is this summary for?

People making decisions concerning public disclosure of patient care performance data.

This summary includes:

  • Key findings from research based on a systematic review
  • Considerations about the relevance of this research for low income countries

Not included:

  • Recommendations
  • Additional evidence not included in the systematic review
  • Detailed descriptions of interventions or their implementation

This summary is based on the following systematic review:

Fung CH, Lim YW, Mattke S, et al. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med 2008; 148:111-23.

 

What is a systematic review?

A summary of studies addressing a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise the relevant research, and to collect and analyse data from the included studies.

SUPPORT – was an international project to support the use of policy relevant reviews and trials to inform decisions about maternal and child health in low- and middle-income countries, funded by the European Commission (FP6) and the Canadian Institutes of Health Research.

Glossary of terms used in this report

Background references on this topic

Background

Public disclosure of performance data has been proposed as a mechanism for improving the quality of care by providing more transparency and by making healthcare providers more accountable. It is assumed that such public disclosures would lead patients to seek care from betterperforming healthcare providers and help healthcare providers to identify areas in which they are deficient and thus improve their performance. Examples of publicly disclosed performance data include mortality rates, patient satisfaction, length of stay, and immunisation coverage.

How this summary was prepared

After searching widely for systematic reviews that can help inform decisions about health systems, we have selected ones that provide information that is relevant to low-income countries. The methods used to assess the reliability of the review and to make judgements about its relevance are described here

Knowing what’s not known is important

A reliable review might not find any studies from low-income countries or might not find any well-designed studies. Although that is disappointing, it is important to know what is not known as well as what is known.

A lack of evidence does not mean a lack of effects. It means the effects are uncertain. When there is a lack of evidence, consideration should be given to monitoring and evaluating the effects of intervention, if it is used.



About the systematic review underlying this summary

Review objectives: To synthesise the evidence for using public disclosure of performance data to improve healthcare quality
Type of What the review authors searched for What the review authors found

Study designs & interventions

Peer-reviewed articles published between 1986 and 2006. Type of studies not pre-specified

2 randomised trials, 2 non-randomised trials, 1 controlled before-after study, 9 interrupted time series studies, and 31 other observational studies.

Participants

Not pre-specified

Hospitals, patients, and hospital staff (45 studies)

Settings

Not pre-specified

USA (43 studies), United Kingdom (1), Canada (1)

Outcomes

Selection of health plans, hospitals, and individual providers, quality improvement activity, clinical outcomes, unintended consequences

Selection of health plans (8 studies), selection of hospitals (9), selection of individual providers (7), quality improvement activity (11), clinical outcomes (11), unintended consequences (13)

Date of most recent search: March 2006

Limitations: Only peer-reviewed, English-language articles were included.
Fung CH, Lim YW, Mattke S, et al. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med 2008; 148:111-23.

Fung CH, Lim YW, Mattke S, et al. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med 2008; 148:111-23.

 

About the certainty of the 
evidence (GRADE)


High: This research provides a very good indication of the likely effect. The likelihood that the effect will be substantially different† is low.


Moderate: This research provides a good indication of the likely effect. The likelihood that effect will be substantially different† is moderate.


Low: This research provides some indication of the likely effect. Howevver, the likelihood that it will be substantially different† is high.


Very low: This research does not provide a reliable indication of the likely effect. The likelihood that the effect will be substantially different† is very high.

* This is sometimes referred to as 'quality of evidence' or 'confidence in the estimate'.

† Substantially different = a large enough difference that it might affect a decision.

See last page for more information.

Summary of findings

The review identified 45 studies conducted in high-income countries. The results are organised by reporting level (health plans, hospitals, individual providers).

1) Health plans

Ten studies assessed the effects of public disclosure of performance data on health plans and showed that:

  • Public disclosure may lead patients to select health plans with better quality ratings or to avoid those with worse ratings. The certainty of this evidence is low.
  • The impact of public disclosure on quality improvement activities is uncertain. This outcome was not reported.
  • Public disclosure may lead to slight improvements in clinical outcomes. The certainty of this evidence is low.

Effects of public disclosure on health plan quality

People:  Users of health insurance plans; health insurance plans
Settings:  USA (10 studies)
Intervention: Public release of performance data
Comparison: No intervention
Outcomes Impact Certainty of the evidence
(GRADE)

Selection of health plans

2 studies found no effect of public disclosure on health plan choices, 6 studies found that public disclosure may lead users to select better-rated plans or to avoid lower-scoring ones.


Quality
improvement activity

Quality improvement activity

No studies of health plan performance data and quality improvement activity were identified.


 -

Clinical outcomes

1 study reported that technical performance measures and patient experience measures were higher for health plans with publicly disclosed data.


GRADE: GRADE Working Group grades of evidence (see above and last page)

 

2) Hospitals

Eighteen studies assessed the effects of the public disclosure of performance data on hospitals and showed that such public reporting:

  • May lead to little or no difference in patient selection of hospitals. The certainty of this evidence is low.
  • Probably stimulates hospitals to undertake quality improvement activities. The certainty of this evidence is moderate.
  • May lead to slight improvements in clinical outcomes. The certainty of this evidence is low.

 

Effects of public disclosure on quality of care provided in hospitals  

People:  Patients, hospitals, hospital staff
Settings:  USA (16 studies), UK (1), Canada (1)
Intervention: Public release of performance data
Comparison: No intervention
Outcomes Impact Certainty of the evidence
(GRADE)

Selection of hospitals

 

3 studies reported that hospitals with better outcomes experienced higher rates of growth in market share, but 6 studies found no association.

 

 

Quality improvement activity

 

9 studies reported that public disclosure led to quality improvement activities; however, 2 studies showed only minimal impact on quality improvement activities.

 

 

Clinical outcomes

 

7 studies found decreases in mortality or improvement in other clinical outcomes; however, 4 studies did not find any impact on clinical outcomes.

 

 
GRADE: GRADE Working Group grades of evidence (see above and last page)

 

3) Individual healthcare providers

Eight studies assessed the effects of public disclosure of performance data on individual providers and showed that:

  • Public disclosure probably influences users of healthcare services to select providers with better quality ratings or to avoid those with worse ratings. The certainty of this evidence is moderate.
  • The impact of public disclosure on quality improvement activities among individual providers of healthcare is uncertain. This outcome was not reported.
  • Public disclosure of performance data may improve clinical outcomes among individual providers. The certainty of this evidence is low.

Effects of public disclosure on quality of care provided by individual healthcare providers

People:  Patients, physicians, or physician groups
Settings:  USA (8 studies)
Intervention: Public disclosure of performance data
Comparison: No intervention
Outcomes Impact Certainty of the evidence
(GRADE)

Selection of providers

6 studies reported that public disclosure affected selection of health providers, while 1 study found no association.

 

Quality improvement activity

No published studies of the effect of public disclosure of performance data on quality improvement activity among physicians or physician groups were identified by the authors.

 

 -

Clinical outcomes

1 study showed that risk-adjusted mortality rates for surgeons decreased after the data were released.

 

 
GRADE: GRADE Working Group grades of evidence (see above and last page)

 

 

 

 

Relevance of the review for low-income countries

 

 

www.supportsummaries.org/methods 

Findings Interpretation*
APPLICABILITY

The studies, all conducted in high-income countries, provided limited evidence of the benefits of public disclosure of performance data.

 

  • Public disclosure of performance data may be difficult to implement in low-income countries because of limitations of the ability of health facilities and providers to produce accurate data, the capacity to disseminate the data, the ability of patients to interpret the data and, in some places, patients not having multiple facilities or providers from which to choose. Consequently, the potential benefits are even more uncertain in low-income countries than in high-income countries.

 

 
EQUITY

In the post-report period of one study, it was noted that patients from more affluent neighborhoods were more likely to be treated by doctors with better quality ratings. Patients from poorer neighborhoods were more likely to be treated by doctors with worse ratings.

 

 

  • This limited evidence suggests either that the public disclosure of performance data may lead wealthier patients to select better performing doctors, that better doctors may charge higher fees which poor patients can’t afford, or both. Such situations may exacerbate healthcare inequalities.

 
ECONOMIC CONSIDERATIONS

None of the studies examined the costs or costeffectiveness of publicly disclosing performance data.

 

 

  • Both costing studies and cost-effectiveness studies are needed, in addition to more rigorus evaluations of the effects of public disclosure of performance data.

 
MONITORING & EVALUATION

There is important uncertainty about the effects of public disclosure of performance data, especially in low-income countries.

 

  • Randomised trials of public disclosure of performance data are needed in low-income countries before scaling up its use or, interrupted time series analyses should be planned and designed before implementing policies that require public disclosure of performance data.

*Judgements made by the authors of this summary, not necessarily those of the review authors, based on the findings of the review and consultation with researchers and policymakers in low-income countries. For additional details about how these judgements were made see:

www.supportsummaries.org/methods

Additional information

Related literature

Faber M, Bosch M, Wollersheim H, et al. Public reporting in health care: how do consumers use quality-of-care information? A systematic review. Med Care 2009; 47:1-8.

 

Henderson A, Henderson S. Provision of a surgeon's performance data for people considering elective surgery. Cochrane Database of Systematic Reviews 2010, Issue 11. Art. No.: CD006327.

 

Ketelaar NABM, Faber MJ, Flottorp S, et al. Public release of performance data in changing the behaviour of healthcare consumers, professionals or organisations. Cochrane Database of Systematic Reviews 2011, Issue 11. Art. No.: CD004538.

 

Kolstad JT, Chernew ME. Quality and consumer decision making in the market for health insurance and health care services. Medical Care Research and Review 2009 ;66(Suppl 1):28S-52S.

 

This summary was prepared by

Charles Shey Wiysonge, Centre for Evidence-based Health Care, Stellenbosch University, & Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa

 

Conflict of interest

None declared. For details, see: www.supportsummaries.org/coi

 

Acknowledgements

This summary has been peer reviewed by: Constance H. Fung, Hanna Bergman, and a reviewer who

prefers to remain anonymous

 

This review should be cited as

Fung CH, Lim YW, Mattke S, et al. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med 2008; 148:111-23.

 

The summary should be cited as

Wiysonge CS. Does public disclosure of performance data improve quality of healthcare? A SUPPORT Summary of a systematic review. August 2016. www.supportsummaries.org

 

Keywords

evidence-informed health policy, evidence-based, systematic review, health systems research, health care, low and middle-income countries, developing countries, primary health care, public release of performance data

 

This summary was prepared with additional support from:

The University of Cape Town, South Africa

The University of Cape Town aspires to become a premier academic meeting point between South Africa, the rest of Africa, and the world. Taking advantage of expanding global networks and our distinct vantage point in Africa, we are committed, through innovative research and scholarship, to grapple with the key issues of our natural and social worlds.

www.uct.ac.za

 



About certainty of evidence (GRADE)

The “certainty of the evidence” is an assessment of how good an indication the research provides of the likely effect, i.e. the likelihood that the effect will be substantially different from what the research found. By “substantially different” we mean a large enough difference that it might affect a decision. These judgements are made using the GRADE system, and are provided for each outcome. The judgements are based on the study design (randomised trials versus observational studies), factors that reduce the certainty (risk of bias, inconsistency, indirectness, imprecision, and publication bias) and factors that increase the certainty (a large effect, a dose response relationship, and plausible confounding). For each outcome, the certainty of the evidence is rated as high, moderate, low or very low using the definitions on page 3.

For more information about GRADE

SUPPORT collaborators:

The Cochrane Effective Practice and Organisation of Care Group (EPOC) is part of the Cochrane Collaboration. The Norwegian EPOC satellite supports the production of Cochrane reviews relevant to health systems in low- and middle-income countries . www.epocoslo.cochrane.org

The Evidence-Informed Policy Network (EVIPNet) is an initiative to promote the use of health research in policymaking in low- and middle-income countries. www.evipnet.org

The Alliance for Health Policy and Systems Research (HPSR) is an international collaboration that promotes the generation and use of health policy and systems research in low- and middle-income countries. www.who.int/alliance-hpsr

Norad, the Norwegian Agency for Development Cooperation, supports the Norwegian EPOC satellite and the production of SUPPORT Summaries. www.norad.no

The Effective Health Care Research Consortium is an international partnership that prepares Cochrane reviews relevant to low-income countries. www.evidence4health.org

To receive e-mail notices of new SUPPORT summaries or provide feedback on this summary, go to: www.supportsummaries.org/contact

Comments