A Comparison of Bayesian Methods for Profiling Hospital Performance
Abstract
There is a growing interest in the use of Bayesian methods for profiling institutional performance. In the literature, several studies have compared different frequentist methods for classifying hospitals as performance outliers. The purpose of this study was to compare 4 different Bayesian methods for classifying hospitals as outcomes outliers, using 30-day hospital-level mortality rates for a cohort of acute myocardial infarction patients as a test case. The 1st Bayesian method involved determining the probability that a hospital’s mortality rate for an average patient exceeded a specified threshold. The 2nd method involved ranking hospitals according to their mortality rate for an average patient. The 3rd method involved determining the probability that a hospital’s standardized mortality ratio exceeded a specified threshold. The 4th method involved ranking hospitals according to their standardized mortality ratio. In most of the scenarios examined, there was only marginal agreement between the different methods. In only 4 of 19 comparisons, was there good agreement between the different methods (0.40 kappa 0.75). Methods based on ranking institutions were relatively insensitive to differences between hospitals. These inconsistencies raise questions about the choice of methods for classifying hospital performance, and they suggest a need for urgent research into which methods are best able to discriminate between institutions and which are most meaningful to decision makers.
Get full access to this article
View all access and purchase options for this article.
References
Luft HS, Romano PS, Remy LL, Rainwater J. Annual Report of the California Hospital Outcomes Project. Sacramento: California Office of Statewide Health Planning and Development, 1993.
Pennsylvania Health Care Cost Containment Council. Focus on Heart Attack in Pennsylvania. Research Methods and Results. Harrisburg: Pennsylvania Health Care Cost Containment Council, 1996.
Tu JV, Austin P, Naylor CD, Iron K, Zhang H. Acute myocardial infarction outcomes in Ontario. In: Naylor CD, Slaugher PM, eds. Cardiovascular Health Services in Ontario: An ICES Atlas. Toronto, Canada: Institute for Clinical Evaluative Sciences, 1999. p. 83-110.
New York State Department of Health. Coronary Artery Bypass Graft Surgery in New York State 1989-1991. Albany: Author, 1992.
Chassin MR, Hannan EL, DeBuono BA. Benefits and hazards of reporting medical outcomes publicly. N Engl J Med. 1996;334:394-398.
Zinman D. State takes docs’ list to heart. Newsday, 18 December 1991. p. 7.
Iezzoni LI. Risk Adjustment for Measuring Health Care Outcomes. Ann Arbor, MI: Health Administration Press, 1994.
Iezzoni LI. Using risk-adjusted outcomes to assess clinical practice: an overview of issues pertaining to risk adjustment. Ann Thorac Surg. 1994;58:1822-1826.
Normand ST, Morris CN, Fung KS, McNeil BJ, Epstein AM. Development and validation of a claims based index for adjusting for risk of mortality: the case of acute myocardial infarction. J Clin Epidemiol. 1995;48:229-243.
Mueller HS, Cohen LS, Braunwald E, et al. Predictors of early morbidity and mortality after thrombolytic therapy of acute myocardial infarction: analyses of patient subgroups in the Thrombolysis in Myocardial Infarction (TIMI) Trial, Phase II. Circulation. 1992;85: 1254-1264.
Volpi A, De Vita C, Franzosi MG, et al. Determinants of 6-month mortality in survivors of myocardial infarction after thrombolysis: results of the GISSI-2 data base. Circulation. 1993;88:416-429.
Dubois C, Pierard LA, Albert A, et al. Short-term risk stratification at admission based on simple clinical data in acute myocardial infarction. Am J Cardiol. 1988;61:216-219.
Suarez C, Herrera M, Vera A, Torrado E, Ferriz J, Arboleda JA. Prediction on admission of in-hospital mortality in patients older than 70 years with acute myocardial infarction. Chest. 1995;108:83-88.
The Multicenter Postinfarction Research Group. Risk stratification and survival after myocardial infarction. N Engl J Med. 1983;309:331-336.
Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan Y. Predicting who dies depends on how severity is measured: implications for evaluating patient outcomes. Ann Intern Med. 1995;123:763-770.
Iezzoni LI, Shwartz M, Ash AS, Hughes JS, Daley J, Mackiernan Y. Severity measurement methods and judging hospital death rates for pneumonia. Med Care. 1996;34:11-28.
Iezzoni LI, Shwartz M, Ash AS, Mackiernan Y. Predicting in hospital mortality for stroke patients. Med Decis Making. 1996; 16:348-356.
Scottish Office. Clinical Outcome Indicators, 1994. Edinburgh: Clinical Resource and Audit Group, 1994.
DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. Comparing risk-adjustment methods for provider profiling. Stat Med. 1997;16:2645-2664.
Leyland AH, Boddy FA. League tables and acute myocardial infarction. Lancet. 1998;351:555-558.
Normand SL, Glickman ME, Gatsonis CA. Statistical methods for profiling providers of medical care: issues and applications. J Am Stat Assoc. 1997;92:803-814.
Christiansen CL, Morris CN. Improving the statistical approach to health care provider profiling. Ann Intern Med. 1997;127:764-768.
Marshall EC, Spiegelhalter DJ. Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates. Br Med J. 1998;316:1701-1705.
Goldstein H, Spiegelhalter DJ. League tables and their limitations: statistical issues in comparisons of institutional performance. J R Stat Soc A. 1996;159:385-443.
Austin PC, Naylor CD, Tu JV. A comparison of a Bayesian versus a frequentist method for profiling hospital performance. J Eval Clin Pract. 2001;7:35-45.
Morris CN, Christiansen CL. Hierarchical models for ranking and identifying extremes, with applications. In: Bernardo JM, Berger JO, Dawid AP, Smith AFM, eds. Bayesian Statistics 5. Oxford, UK: Oxford University Press, 1996. p. 277-296.
Tu JV, Naylor CD, Austin P. Temporal changes in the outcomes of acute myocardial infarction in Ontario, 1992-1996. Can Med Assoc J. 1999;161:1257-1261.
Cox JL, Melady MP, Chen E, Naylor CD. Towards improved coding of acute myocardial infarction in hospital discharge abstracts: a pilot project. Can J Cardiol. 1997;13:351-358.
Tu JV, Austin PC, Walld R, Roos L, Agras J, McDonald KM. Development and validation of the Ontario Acute Myocardial Infarction Mortality Prediction Rules. J Am Coll Cardiol. 2001;37:992-997.
Wong G, Mason W. The hierarchical logistic regression model for multilevel analysis. J Am Stat Assoc. 1985;80:513-524.
Spiegelhalter DJ, Best NG, Gilks WR, Inskip H. Hepatitis B: a case study in MCMC methods. In: Gilks WR, Richardson S, Spiegelhalter DJ, eds. Markov Chain Monte Carlo in Practice. London: Chapman & Hall, 1996. p. 21-43.
Gilks WR, Richardson S, Spiegelhalter DJ. Introducing Markov chain Monte Carlo. In: Gilks WR, Richardson S, Spiegelhalter DJ, eds. Markov Chain Monte Carlo in Practice. London: Chapman & Hall, 1996. p. 1-19.
Gilks WR, Thomas A, Spiegelhalter DJ. A language and program for complex Bayesian modelling. Statistician. 1994;43:169-178.
Geweke J. Evaluating the accuracy of sampling-based approaches to calculating posterior moments. In: Bernardo JM, Berger JO, Dawid AP, Smith AFM, eds. Bayesian Statistics 4. Oxford, UK: Clarendon Press, 1992. p. 169-193 (with discussion).
Heidelberger P, Welch P. Simulation run length control in the presence of an initial transient. Operations Res. 1983;31:1109-1144.
Cicchetti DV, Allison T. A new procedure for assessing reliability of scoring EEG sleep recordings. Am J EEG Technol. 1971;11:101-109.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published: April 2002
Issue published: April 2002
Keywords
PubMed: 11958498
Authors
Metrics and citations
Metrics
Article usage*
Total views and downloads: 140
*Article usage tracking started in December 2016
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 46 view articles Opens in new tab
Crossref: 49
- Evaluating methods of outlier detection when benchmarking clinical registry data–a simulation study
- Variation in First-line Type 2 Diabetes Treatment due to eGFR and Provider Preferences: A Novel Statistical Analysis
- Change in Hospital Risk-Standardized Stroke Mortality Performance With and Without the Passive Surveillance Stroke Severity Score
- Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review
- Hospital Profiling Using Bayesian Decision Theory
- The failure of four bootstrap procedures for estimating confidence intervals for predicted-to-expected ratios for hospital profiling
- A new standardized mortality ratio method for hospital quality evaluation
- Examining Hospital Variation on Multiple Indicators of Stroke Quality of Care
- A pilot study to assess the learning environment and use of reliability enhancing work practices in VHA cardiac catheterization laboratories
- Comparing a multivariate response Bayesian random effects logistic regression model with a latent variable item response theory model for provider profiling on multiple binary indicators simultaneously
- View More
Figures and tables
Figures & Media
Tables
View Options
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
loading institutional access options
SMDM members can access this journal content using society membership credentials.
SMDM members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
