Research article | Open | Open Peer Review | Published:
Variation in rates of ICU readmissions and post-ICU in-hospital mortality and their association with ICU discharge practices
BMC Health Services Researchvolume 17, Article number: 281 (2017)
Variation in intensive care unit (ICU) readmissions and in-hospital mortality after ICU discharge may indicate potential for improvement and could be explained by ICU discharge practices. Our objective was threefold: (1) describe variation in rates of ICU readmissions within 48 h and post-ICU in-hospital mortality, (2) describe ICU discharge practices in Dutch hospitals, and (3) study the association between rates of ICU readmissions within 48 h and post-ICU in-hospital mortality and ICU discharge practices.
We analysed data on 42,040 admissions to 82 (91.1%) Dutch ICUs in 2011 from the Dutch National Intensive Care Evaluation (NICE) registry to describe variation in standardized ICU readmission and post-ICU mortality rates using funnel-plots. We send a questionnaire to all Dutch ICUs. 75 ICUs responded and their questionnaire data could be linked to 38,498 admissions in the NICE registry. Generalized estimation equations analyses were used to study the association between ICU readmissions and post-ICU mortality rates and the identified discharge practices, i.e. (1) ICU discharge criteria; (2) bed managers; (3) early discharge planning; (4) step-down facilities; (5) medication reconciliation; (6) verbal and written handover; (7) monitoring of post-ICU patients; and (8) consulting ICU nurses. In all analyses, the outcomes were corrected for patient-related confounding factors.
The standardized rate of ICU readmissions varied between 0.14 and 2.67 and 20.8% of the hospitals fell outside the 95% control limits and 3.6% outside the 99.8% control limits. The standardized rate of post-ICU mortality varied between 0.07 and 2.07 and 17.1% of the hospitals fell outside the 95% control limits and 4.9% outside the 99.8% control limits. We could not demonstrate an association between the eight ICU discharge practices and rates of ICU readmissions or post-ICU in-hospital mortality. Implementing a higher number of ICU discharge practices was also not associated with better patient outcomes.
We found both variation in patient outcomes and variation in ICU discharge practices between ICUs. However, we found no association between discharge practices and rates of ICU readmissions or post-ICU mortality. Further research is necessary to find factors, which may influence these patient outcomes, in order to improve quality of care.
Intensive care unit (ICU) readmissions pose an important clinical problem because they are associated with patient harm, inefficiencies and higher costs [1–4]. Patients readmitted to the ICU experience more adverse events, with in-hospital mortality rates up to six times higher than non-readmitted patients . Readmitted patients reduce ICU bed availability and it is possible that ICU facilities could be used more efficiently if ICU readmissions could be prevented [1–4].
Risk factors for ICU readmission and in-hospital mortality following ICU discharge include patient characteristics, such as age, co-morbidities and severity of illness [5, 6], and organisational factors, such as discharge time and the availability of step-down facilities [5–8]. A substantial amount of variation in patient outcomes between hospitals may be explained by the organisation of the ICU discharge process , which consists of four essential components: decision making, planning and preparation, patient transport and follow-up. The ICU discharge process is a complex process in which many healthcare professionals are involved . Deficits in communication, coordination of care, and information exchange between ICU and general ward professionals [11–13] may increase the risk of a suboptimal handover, severe adverse events, ICU readmissions and mortality . Patients discharged from the ICU are particularly vulnerable to poor handovers due to the complicated physiology  and the substantial decrease in monitoring when these patients are transferred from the ICU to a general ward [16, 17].
Several methods and instruments are available which aim to improve the quality of the discharge of ICU patients to general wards, such as a liaison nurse and handover forms . Evidence of the effectiveness of these interventions, however, is limited [18, 19] and the actual use of ICU discharge practices vary between ICUs .
Variation in ICU readmissions and in-hospital mortality after ICU discharge between hospitals may indicate potential for improvement and be explained by the ICU discharge practices which have been implemented. Insight into associations between ICU discharge practices and patient outcomes can provide evidence for professionals on ways to improve their ICU discharge process, and possibly, reduce adverse patient outcomes.
The aims of this study were: (1) to describe variation in rates of ICU readmissions within 48 h and post-ICU in-hospital mortality in individual hospitals; (2) to describe current ICU discharge practices in Dutch hospitals; and (3) to study the association between ICU discharge practices and rates of ICU readmissions within 48 h and post-ICU in-hospital mortality. We hypothesized that the implementation of ICU discharge practices would be associated with lower rates of ICU readmissions and lower rates of post-ICU in-hospital mortality.
The design of the study was pre-specified and published .
Patient data and outcomes
The Dutch National Intensive Care Evaluation (NICE) registry collects demographic, physiological, clinical and organizational data from ICUs. To ensure that the data are of a high quality, ICU employees are trained in how to score patients, the data are checked before being read into the database, and data quality audits are carried out [22–25].
We used data from the NICE registry on ICU admissions, for reasons other than cardiac surgery, between 1st January and 31st December 2011. We did not examine admissions following cardiac surgery, because cardiac surgery is only performed in a small number of hospitals in the Netherlands and these patients have a low risk of ICU readmission or post-ICU in-hospital mortality [1, 26]. We excluded admissions, in which the patient died during the initial ICU admission or was discharged from the ICU and hospital simultaneously, because these patients were not at risk for ICU readmission or post-ICU in-hospital mortality. We also excluded admissions not fulfilling the Acute Physiology and Chronic Health Evaluation (APACHE) IV inclusion criteria  and with missing data on type of admission, reason for discharge, APACHE physiology score, APACHE reason for admission or discharge location (Additional file 1).
We defined an initial ICU admission as a patient’s first ICU admission within a single hospital stay and an ICU readmission as the first ICU readmission within 48 h of the initial ICU discharge, but within the same hospital stay. We choose a time frame of 48 h, as readmissions within this period have a stronger relationship with ICU interventions, such as mechanical ventilation, and discharge circumstances, than later readmissions . We defined post-ICU in-hospital mortality as the death of the patient after the initial ICU admission ended, but before he or she was discharged from the hospital.
ICU discharge practices
Members of an expert panel, consisting of one internal medicine consultant, two intensive care consultants and two researchers, selected eight ICU discharge practices described in scientific literature and clinical guidelines [18, 19, 28–36] to examine in this study. We present these eight practices in Table 1. They were the use of: (1) ICU discharge criteria [29, 30]; (2) a bed manager [31, 32]; (3) early discharge planning  (4) step-down facilities [28, 30]; (5) medication reconciliation [32, 34]; (6) verbal and written handover [28, 30, 32]; (7) monitoring of post-ICU patients ; and (8) consulting ICU nurses . We extracted data on the use of step-down facilities from the NICE registry. We collected data on the use of the other seven ICU discharge practices using an online questionnaire (Additional file 2), sent to all Dutch ICUs in May 2012. We sent reminders after 9 days and after 3 weeks and contacted the non-responding ICUs by telephone a month after initially sending out the questionnaire. We transformed the data on the use of the eight discharge practices into dichotomous variables to indicate the presence or absence of a discharge practice on a specific ICU (Additional file 3). We summed the eight dichotomous variables into a combined practice score, representing the number of discharge practices incorporated into the discharge process in each ICU.
We calculated the standardised readmission and post-ICU mortality rates for each hospital by dividing the observed number of readmissions or deaths by the expected number of readmissions or deaths. The expected number of readmissions or deaths was the sum of the predicted probabilities of readmission or death obtained from separate prediction models. Readmission rates were corrected for ICU level (in which level 1 are the least and level 3 the most advanced ICUs), age, cardiovascular insufficiency, cirrhosis, hematological malignancy, cardio vascular accident, medical or surgical admission type, planned admission, mechanical ventilation in the first 24 h of admission, chronic renal insufficiency, chronic dialysis, chronic obstructive pulmonary disease, respiratory insufficiency, neoplasm, immunological insufficiency, gastrointestinal bleeding, acute renal failure, confirmed infection, vasopressors, and logit transformed APACHE IV mortality probability . Mortality rates were corrected for ICU level (in which level 1 are the least and level 3 the most advanced ICUs), age, cardiovascular insufficiency, cirrhosis, hematological malignancy, cardio vascular accident, medical or surgical admission type, planned admission, mechanical ventilation in the first 24 h of admission, chronic renal insufficiency, chronic dialysis, chronic obstructive pulmonary disease, respiratory insufficiency, neoplasm, immunological insufficiency, gastrointestinal bleeding, acute renal failure, confirmed infection, vasopressors, diabetes, cerebrovasculair accident, CPR, dysrhythmia, and logit transformed APACHE IV mortality probability . We assessed the discrimination of the prediction models using the area under the receiver operating characteristic (ROC) curve  and the calibration using the Hosmer-Lemeshow goodness-of-fit statistic Ĉ with 10 groups . We presented the standardized rates in funnel plots with 95 and 99.8% control limits. We obtained the control limits under the assumption that the natural logarithms of the standardized rates follow a normal distribution . ICUs outside the control limits can be interpreted as deviating significantly from the national rates.
We analysed the univariate association between ICU readmission and post-ICU in-hospital mortality and the eight ICU discharge practices using generalized estimation equations with a logit link function and robust variance estimators , while correcting for patient factors. We applied the Bonferroni correction to correct for multiple testing , and hence viewed the association between a ICU discharge practice and ICU readmission or post-ICU in-hospital mortality if p-value < 0.0056 (0.05/9). We performed the statistical analyses using IBM SPSS Statistics and R 2.13.0.
We extracted 59,181 first admissions to ICUs in 82 hospitals from the NICE registry (Fig. 1). We excluded 17,141 (Additional file 1) and included 42,040 admissions (71.0%) when calculating standardised readmission and post-ICU mortality rates. The ICUs were in six (7.3%) university hospitals, 29 (35.4%) teaching hospitals, and 47 (57.3%) general hospitals. We present the patient characteristics in Table 2.
Rates of ICU readmissions and post-ICU in-hospital mortality
We found a crude ICU readmission rate of 2.9% (1,216/42,040). The standardized rates varied between 0.14 and 2.7 with, by definition, an overall target rate of 1.00. In Fig. 2, we present a funnel plot of the standardized rates of ICU readmissions against the number of ICU admissions per ICU in 2011. In total, 65 (79.3%) ICUs fall within the 95% control limits, three (3.7%) above the upper, and 14 (17.1%) below the lower 95% control limits. One (1.2%) hospital falls above the upper and two (2.4%) hospitals fall below the lower 99.8% control limits. The calibration (Ĉ = 18.1, p-value = 0.0205) and discrimination (area under the ROC curve = 0.63) of the standardization model for ICU readmissions were poor.
We found a crude hospital mortality rate of 6.7% (2,811/42,040). The standardized rates ranged between 0.1 and 2.1. In Fig. 3, we present a funnel plot of these rates against the number of ICU admissions in 2011. Five hospitals (6.1%) have an adjusted post-ICU in-hospital mortality rate above the upper and nine (11.0%) below the lower 95% control limits. Four hospitals (4.9%) fall below the lower 99.8% control limits. Although the discrimination (area under the ROC curve = 0.82) of the standardization model for post-ICU in-hospital mortality was good, the calibration (Ĉ = 38.9, p-value < 0.0001) was poor.
ICU discharge practices, ICU readmission and post-ICU in-hospital mortality
We had data from the NICE registry and a completed questionnaire for 75 ICUs (Additional file 4). To study the association between ICU discharge practices and ICU readmissions and post-ICU mortality, we excluded 3542 admissions to the non-participating ICUs. Hence, we used data on 38,498 admissions (65.1%) to 75 ICUs in the analyses on associations between ICU discharge practices and ICU readmission and post-ICU in-hospital mortality (Fig. 1). These ICUs were in six university hospitals (8.0%), 28 tertiary medical teaching hospitals (37.3%), and 41 general hospitals (54.7%). We present the results from the questionnaire in detail in Additional file 4.
Table 3 shows the percentages of ICUs, which had implemented each of the discharge practices, and the odds ratios of the univariate association between the implementation of each discharge practice and ICU readmission and post-ICU in-hospital mortality. Following the Bonferroni correction (p < (0,05/9) = p < 0.0056), none of the ICU discharge practices nor the total number of ICU discharge practices implemented by each ICU were associated with the standardized rates of readmission or mortality.
The objective of our study was to describe variation in ICU readmissions within 48 h and post-ICU in-hospital mortality and to study the association of these patient outcomes with the implementation of ICU discharge practices. Using funnel plots, we found that 20.8% of the ICUs fell outside the 95% control limits and 3.6% outside the 99.8% control limits with respect to ICU readmission and 17.1 and 4.9% with respect to post-ICU in-hospital mortality. The substantial proportion of ICUs with standardized readmission or mortality rates falling outside the control limits, suggests that there is more variation between hospitals on these patient outcomes than would be expected and that there is room for quality improvement. The ICUs with lower readmission or mortality can be viewed as best practices and ICUs with higher readmission or mortality could benefit from their views and experiences. The extent of this variation is consistent with that reported in studies on ICU length of stay and mortality [42, 43]. To study and to possibly explain the found variation, we subsequently studied the implementation of ICU discharge practices and their association with the occurrence of IC readmissions and post-ICU in hospital mortality rate.
In this study, we also found that ICU discharge practices vary. We had hypothesized that such variation could indicate that the ICU discharge process could be optimized and, hence, potentially improve patient outcomes and reduce healthcare costs [44, 45]. However, we were unable to demonstrate an association between ICU discharge practices and rates of ICU readmissions or post-ICU in-hospital mortality. In addition, implementing a higher number of ICU discharge practices was not associated with better patient outcomes. Results of previous studies reporting about the association between the use of patient safety and quality improvement practices, and patient safety outcomes are diffuse. Some studies showed that compliance to discharge practices was associated with lower hospital complications and mortality rates [46, 47], while others showed no association .
An important strength of this study is our large dataset covering more than 90% of all Dutch ICUs. In addition, we included the APACHE IV mortality probability in our case-mix correction models. Currently, the APACHE IV is the best performing model for case-mix correction for in-hospital mortality following ICU admission in the Netherlands [25, 26].
Our study has some limitations. We strived to minimize the effects of case-mix differences between ICUs by presenting case-mix adjusted standardized rates for quantifying variation in patient outcomes. However, our case-mix correction models have not been externally validated and, in our dataset, the calibration of the models for ICU readmission and post-ICU mortality and the discrimination of the model for ICU readmission were poor. This means that these models may not adequately correct for case-mix differences between hospitals, potentially resulting in more hospitals than expected falling outside the control limits . In addition, the variation between the rates of ICU readmission and post-ICU in-hospital mortality may still result from chance , overdispersion [39, 51], an incorrect method for determining the control limits , or registration problems within the hospitals.
The number of admissions included in our analyses for some ICUs was very low. Therefore, even when using funnel plots, there is a low probability of detecting that these ICUs are performing differently from national rates .
We found no significant association between ICU discharge practices and patient outcomes which may be due to several limitations of our study. First, the power to detect a reduction in post-ICU mortality and ICU readmission rate was limited because we measured each of the discharge practices at hospital level and, although the response rate to the questionnaire was 91.5%, the number of ICUs was limited. Furthermore, some practices were present in almost every ICU. Second, the use of discharge practices were measured using a self-reported questionnaire, which may be susceptible to bias. Overestimation of own practices and socially desirable answers could have influenced our findings. However, Scholle and colleagues found only minor overestimation in their study and concluded that self-assessment could be useful for quality improvement purposes . Third, in our regression models we used patient data, such as severity of illness and the APACHE IV reasons for ICU admission, measured at the time of ICU admission. Ideally, data representing the patient’s condition at the time of ICU discharge would be used. However, these data are not available in the NICE registry. Fourth, we had no data on whether patients were discharged from the ICU for palliative care on the ward. This could have led to an overestimation of the mortality rates.
Clinical handover has been identified as a key process in improving quality of care and patient safety and reducing adverse patient outcomes [55, 56]. Quantification of variation is a tool for uncovering suboptimal quality of care and may identify potential for improvement [57–59]. We found both variation in patient outcomes and in discharge practices and reasoned that this indicates potential for improving patient outcomes and subsequently, reducing healthcare costs [44, 45]. However, we were not able to identify a relation between ICU discharge practices and patient outcomes. Further research is necessary to find factors, which may influence these patient outcomes, in order to improve quality of care. For example organisational factors, such as staffing and experience and skills of (ICU) personnel. Unfortunately, we were not able to include them in our research due to the lack of data of these factors. Exploratory research into the differences between the hospitals falling above the upper and below the lower control limits in our funnel plots may give insight into factors influencing quality of care.
Causes of ICU readmissions and post-ICU in-hospital mortality are likely to vary between hospitals. Although interventions to reduce the rates of these events have been described in the literature, our study shows that none of them are associated with better outcomes in the Netherlands. Examining individual ICU readmissions or post-ICU in-hospital mortalities locally may provide ICUs insight into potential areas for improvement in their own ICU discharge process.
Acute physiology and chronic health evaluation
Intensive care unit
National intensive care evaluation
Receiver operating characteristic
Kramer AA, Higgins TL, Zimmerman JE. The association between ICU readmission rate and patient outcomes. Crit Care Med. 2013;41:24.
Renton J, Pilcher DV, Santamaria JD, Stow P, Bailey M, Hart G, et al. Factors associated with increased risk of readmission to intensive care in Australia. Intensive Care Med. 2011;37:1800.
Nishi GK, Suh RH, Wilson MT, Cunneen SA, Margulies DR, Shabot MM. Analysis of causes and prevention of early readmission to surgical intensive care. Am Surg. 2003;69:913.
Marquet K, Claes N, De Troy E, Kox G, Droogmans M, Schrooten W, et al. One fourth of unplanned transfers to a higher level of care are associated with a highly preventable adverse event: a patient record review in six belgian hospitals. Crit Care Med. 2015;43:1053.
Elliott M, Worrall-Carter L, Page K. Intensive care readmission: a contemporary review of the literature. Intensive Care Nurs. 2014;30:121.
Elliott M, Worrall-Carter L, Page K. Factors associated with in-hospital mortality following ICU discharge: a comprehensive review. British Journal of Intensive Care. 2012;winter:120.
Gantner D, Farley K, Bailey M, Huckson S, Hicks P, Pilcher D. Mortality related to after-hours discharge from intensive care in Australia and New Zealand, 2005–2012. Intensive Care Med. 2014;40:1528.
Town JA, Churpek MM, Yuen TC, Huber MT, Kress JP, Edelson DP. Relationship between ICU bed availability, ICU readmission, and cardiac arrest in the general wards. Crit Care Med. 2014;42:2037.
Zegers M, De Bruijne MC, Spreeuwenberg P, Wagner C, Van Der Wal G, Groenewegen PP. Variation in the rates of adverse events between hospitals and hospital departments. Int J Qual Health Care. 2011;23:126.
Lin F, Chaboyer W, Wallis M. A literature review of organisational, individual and teamwork factors contributing to the ICU discharge process. Aust Crit Care. 2009;22:29.
Hellesø R, Lorensen M, Sorensen L. Challenging the information gap - the patients transfer from hospital to home health care. Int J Med Inform. 2004;73:569.
Perren A, Conte P, De Bitonti N, Limoni C, Merlani P. From the ICU to the ward: cross-checking of the physician’s transfer report by intensive care nurses. Intensive Care Med. 2008;34:2054.
Hesselink G, Schoonhoven L, Barach P, Spijker A, Gademan P, Kalkman C, et al. Improving patient handovers from hospital to primary care: a systematic review. Ann Intern Med. 2012;157:417.
Häggström M, Asplund K, Kristiansen L. Struggle with a gap between intensive care units and general wards. Int J Qual Stud Well-being. 2009;4:181.
Chaboyer W, Thalib L, Foster M, Ball C, Richards B. Predictors of adverse events in patients after discharge from the intensive care unit. Am J Crit Care. 2008;17:255.
Häggström M, Asplund K, Kristiansen L. To reduce technology prior discharge from intensive care - important but difficult? A grounded theory. Scand J Caring Sci. 2013;27:506.
Stelfox HT, Perrier L, Straus SE, Ghali WA, Zygun D, Boiteau P, et al. Identifying intensive care unit discharge planning tools: protocol for a scoping review. BMJ Open. 2013;3:e002653.
van Sluisveld N, Hesselink G, van der Hoeven JG, Westert G, Wollersheim H, Zegers M. Improving clinical handover between intensive care unit and general ward professionals at intensive care unit discharge: a systematic review. Intensive Care Med. 2015;41:589.
Niven DJ, Bastos JF, Stelfox HT. Critical care transition programs and the risk of readmission or death after discharge from an ICU: a systematic review and meta-analysis. Crit Care Med. 2013;42:179.
Heidegger CP, Treggiari MM, Romand JA, Swiss ICU Network. A nationwide survey of intensive care unit discharge practices. Intensive Care Med. 2005;31:1676.
van Sluisveld N, Zegers M, Westert G, van der Hoeven JG, Wollersheim H. A strategy to enhance the safety and efficiency of handovers of ICU patients: study protocol of the pICUp study. Implement Sci. 2013;8:67.
Koetsier A, Peek N, de Jonge E, Dongelmans D, van Berkel G, de Keizer N. Reliability of in-hospital mortality as a quality indicator in clinical quality registries. A case study in an intensive care quality register. Methods Inf Med. 2013;52:432.
DW D l, Dusseljee J, Brinkman S, Van Berkel R, Van Maanen R, Bosman RJ, et al. Severity of illness and outcome in ICU patients in the Netherlands: results from teh NICE registry 2006–2007. Neth J Crit Care. 2009;13:7.
Arts D, de Keizer N, Scheffer GJ, de Jonge E. Quality of data collected for severity of illness scores in the Dutch national intensive care evaluation (NICE) registry. Intensive Care Med. 2002;28:656.
Brinkman S, Bakhshi-Raiez F, Abu-Hanna A, de Jonge E, Bosman RJ, Peelen L, et al. External validation of acute physiology and chronic health evaluation IV in Dutch intensive care units and comparison with acute physiology and chronic health evaluation II and simplified acute physiology score II. J Crit Care. 2011;26:105.
Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med. 2006;34:1297.
Brown SE, Ratcliffe SJ, Kahn JM, Halpern SD. The epidemiology of intensive care unit readmissions in the United States. Am J Respir Crit Care Med. 2012;185:955.
Dutch Healthcare Inspectorate. Grote intensive care-afdelingen werken continu aan kwaliteit: Inspectieonderzoek versnelt resultaten. Dutch Healthcare Inspectorate. 2010. http://www.igz.nl/Images/2011-09-02%20Rapport%20Grote%20IC-afdelingen%20-%20versie%20met%20bijlagen_tcm294-306678.pdf. Accessed 12 Aug 2015.
Buter H. Criteria voor opname en ontslag van Intensive Care afdelingen in Nederland. Nederlandse Vereniging voor Intensive Care. 2011. http://nvic.nl/sites/default/files/Richtlijnen%20aanmaken/Herziene%20richtlijn%20Opname%20en%20ontslag%20criteria%20januari%202011.pdf. Accessed 12 Aug 2015
Commissie Nationale Kwaliteitsvisitatie Intensive Care. Kwaliteitsvisitatie afdeling intensive care. Commissie Nationale Kwaliteitsvisitatie Intensive Care; 2010. http://nvic.nl/commissie-nationale-kwaliteitsvisitatie-intensive-care. Accessed 12 Aug 2015.
South Eastern Sydney Illawarra. Critical Care Bed Management Procedure. NSW Health; 2015. https://www.seslhd.health.nsw.gov.au/Policies_Procedures_Guidelines/Clinical/Critical_Care_Emergency_Medicine/Documents/SESLHDPR228_CriticalCareBedManagemen%20Procedure.pdf. Accessed 12 Aug 2015.
Critical Care Network Northern Ireland. Policy for Discharge from Adult Critical Care Services. Critical Care Network Northern Ireland; 2009. http://www.southerntrust.hscni.net/pdf/DischargeFromAdultCriticalCareServicesPolicy.pdf. Accessed 12 Aug 2015.
Schouten L. Intensive care aanzienlijk verbeterd. Med Contact. 2002;3
Pronovost P, Weast B, Schwarz M, Wyskiel RM, Prow D, Milanovich SN, et al. Medication reconciliation: a practical tool to reduce the risk of medication errors. J Crit Care. 2003;18:201.
Esdonk HGJ. Ruimte op de Intensive Care. Med Contact. 2002;15
Endacott R, Chaboyer W, Edington J, Thalib L. Impact of an ICU Liaison Nurse Service on major adverse events in patients recently discharged from ICU. Resuscitation. 2010;81:198.
Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29.
Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med. 1997;16:965.
Spiegelhalter DJ. Funnel plots for comparing institutional performance. Stat Med. 2005;24:1185.
Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics. 1986;42:121.
Armstrong RA. When to use the Bonferroni correction. Ophthalmic Physiol Opt. 2014;34:502.
Keenan SP, Dodek P, Martin C, Priestap F, Norena M, Wong H. Variation in length of intensive care unit stay after cardiac arrest: where you are is as important as who you are. Crit Care Med. 2007;35:836.
Kuzniewicz MW, Vasilevskis EE, Lane R, Dean ML, Trivedi NG, Rennie DJ, et al. Variation in ICU risk-adjusted mortality: impact of methods of assessment and potential confounders. Chest. 2008;133:1319.
Sinuff T, Muscedere J, Adhikari NK, Stelfox HT, Dodek P, Heyland DK, et al. Knowledge translation interventions for critically ill patients: a systematic review. Crit Care Med. 2013;41:2627.
Berwick DM, Hackbarth AD. Eliminating waste in US health care. JAMA. 2012;307:1513.
Brooke BS, Dominici F, Pronovost PJ, Makary MA, Schneider E, Pawlik TM. Variations in surgical outcomes associated with hospital compliance with safety practices. Surgery. 2012;151:651.
Groene O, Mora N, Thompson A, Saez M, Casas M, Sunol R. Is the maturity of hospitals’ quality improvement systems associated with measures of quality and patient safety? BMC Health Serv Res. 2011;11:344.
Kernisan LP, Lee SJ, Boscardin WJ, Landefeld CS, Dudley RA. Association between hospital-reported leapfrog safe practices scores and inpatient mortality. JAMA. 2009;301:1341.
Cook DA, Duke G, Hart GK, Pilcher D, Mullany D. Review of the application of risk-adjusted charts to analyse mortality outcomes in critical care. Crit Care Resusc. 2008;10:239.
Seaton SE, Manktelow BN. The probability of being identified as an outlier with commonly used funnel plot control limits for the standardised mortality ratio. BMC Med Res Methodol. 2012;12:98.
Spiegelhalter DJ. Handling over-dispersion of performance indicators. Qual Saf Health Care. 2005;14:347.
Manktelow BN, Seaton SE. Specifying the probability characteristics of funnel plot control limits: an investigation of three approaches. PLoS One. 2012;7:e45723.
Seaton SE, Barker L, Lingsma HF, et al. What is the probabilty of detecting poorly performing hospitals using funnel plots? BMJ Qual Saf. 2013;22:870.
Scholle SH, Pawlson LG, Solberg LI, Shih SC, Asche SE, Chou AF, et al. Measuring practice systems for chronic illness care: accuracy of self-reports from clinical personnel. Jt Comm J Qual Patient Saf. 2008;34:407.
Manser T, Foster S. Effective handover communication: an overview of research and improvement efforts. Best Prac Res Clin Anaesthesiol. 2011;25:181.
World Health Organization. Communication during patient hand-overs. Patient Safety Solutions, Volume 1, Solution 3. WHO Press; 2007. http://www.who.int/patientsafety/solutions/patientsafety/PS-Solution3.pdf. Accessed 12 Aug 2015.
The NHS Confederation. Variation in healthcare: does it matter and can anything be done? NHS Confederation. 2004. http://www.nhsconfed.org/~/media/Confederation/Files/Publications/Documents/Variation%20in%20healthcare.pdf. Accessed 12 Aug 2015.
LaPar DJ, Ghanta RK, Kern JA, Crosby IK, Rich JB, Speir AM, et al. Hospital variation in mortality from cardiac arrest after cardiac surgery: an opportunity for improvement? Ann Thorac Surg. 2014;98:534.
Tsai TC, Joynt KE, Orav EJ, Gawande AA, Jha AK. Variation in surgical-readmission rates and quality of hospital care. N Eng J Med. 2013;369:1134.
We would like to thank all ICUs participating in the NICE registry.
This study was funded by ZonMw, the Netherlands Organisation for Health Research and Development (dossier number: 171203006).
Availability of data and materials
The data that support the findings of this study are available from the NICE database but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available.
NS has made substantial contributions to the design, analysis and interpretation of the data and drafted the manuscript. FR has made substantial contributions to the acquisition of data, analysis and interpretation of data, and helped to draft the manuscript. NK has made substantial contributions to the acquisition and the interpretation of data and revised the manuscript critically for important intellectual content. RH has made substantial contributions to the analysis and interpretation of data and revised the manuscript critically for important intellectual content. GW has made substantial contributions to the analysis and interpretation of the data and revised the manuscript critically for important intellectual content. HW has made substantial contribution to conception and design and revised the manuscript critically for important intellectual content. JGH has made substantial contributions to the conception and design, interpretation of the data and revised the manuscript critically for important intellectual content. MZ has made substantial contributions to conception and design, to the analysis and interpretation of the data and helped to draft the manuscript. All authors read and approved the final manuscript.
MZ was supported by a research fellowship sponsored by ZonMw (dossier number 170996006). The NICE foundation pays the department medical informatics for maintaining, processing and analyzing the data in the NICE registry. The funding sources did not participate in study conception, data collection, analyses, manuscript preparation, the decision to submit the manuscript for publication, or any other part of the study. The remaining authors have disclosed that they do not have any potential conflicts of interest.
Consent for publication
Ethics approval and consent to participate
We used an anonymized data set for this study [23–25]. We presented the study protocol  to the medical ethical committee of the Radboud University Medical Center (registration number: 2011/460). This committee stated that ethical approval was not required under Dutch National Law.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Number of admissions excluded per exclusion criterion. (PDF 83 kb)
Questionnaire ‘ICU discharge practice’ translated from Dutch to English. (PDF 112 kb)
Dichotomising Questionnaire variables. (PDF 80 kb)
Results questionnaire. (PDF 99 kb)
Database Questionnaire (XLSX 37 kb)