External validation shows Epic Sepsis Model is a poor predictor of sepsis in hospitalized patients
1. This external validation study suggests that the Epic Sepsis Model, a proprietary sepsis prediction algorithm, was a poor predictor of sepsis in hospitalized patients, with poor sensitivity, discrimination, and calibration in predicting the onset of sepsis.
2. The widespread implementation of the Epic Sepsis Model despite its poor clinical correlation and performance should raise concerns among governing professional medical organizations about sepsis management in hospital.
Evidence Rating Level: 2 (Good)
Study Rundown: The Epic Sepsis Model (ESM) is a proprietary sepsis prediction tool that has been widely adopted and integrated into electronic health record systems at hundreds of hospitals in the United States. This implementation has occurred despite the lack of independent validation of the model, raising concerns regarding sepsis management at a national level. Thus, a detailed evaluation of how well the ESM performs has major implications for informing clinical decision-making for thousands of patients hospitalized for sepsis in the US each year. This cohort study sought to present an independently conducted external validation of the ESM using data from Michigan Medicine, the major academic health system affiliated with the University of Michigan, Ann Arbor. For the analysis, model discrimination was evaluated via area under the curve for receiver operating characteristics at the hospitalization level. Model calibration was assessed via calibration plots and the potential clinical benefits associated with using ESM was validated by comparing the ESM score with contemporary clinical practice, based on timely administration of antibiotics. In addition, alert fatigue was assessed by comparing the clinical impact of various alerting strategies. In total, 27,697 patients undergoing 38,455 hospitalizations met the inclusion criteria, of whom sepsis occurred in 2,552 (7%) of them. The ESM predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance originally reported by the developer. These findings suggest that the ESM is a poor algorithm for predicting sepsis in hospital. Furthermore, governing professional medical associations should be wary of the increasing deployment of such proprietary models and carefully review the broad use of these algorithms and make formal recommendations about their use through national guidelines. A limitation of this study was that data compilation and eventual analysis for external validation of ESM was completed at Michigan Medicine, a single academic medical institution that may not reflect the diversity and demographics of the United States as a nation or be applicable to other populations.
Click to read the study in JAMA Internal Medicine
Click to read an accompanying editorial in JAMA Internal Medicine
Relevant Reading: An electronic tool for the evaluation and treatment of sepsis in the ICU: a randomized controlled trial
In-Depth [retrospective cohort]: This external validation cohort study was conducted among hospitalized patients aged 18 years or older admitted to Michigan Medicine, the academic medical center affiliated with the University of Michigan, Ann Arbor, between December 2018 and October 2019. The authors chose a composite definition of sepsis combining the Centers for Disease Control and Prevention surveillance criteria and International Statistical Classification of Diseases and Related Health Problems, 10th Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another. This composite definition was chosen to account for the most common reasons why health care institutions track sepsis data – namely, surveillance and quality assessment, although alternative definitions of sepsis are still available. The study included 27,697 patients who had 38,455 hospitalizations (21,904 women [57%]; median age, 56 years [IQR, 35-69 years]), of whom 2,552 (7%) developed sepsis in hospital. Using ESM, the area under the receiver operating characteristic curve for hospitalization level was 0.63 (95%CI, 0.62-0.64). The ESM was able to identify 183 of 2,552 patients with sepsis (7%) who did not receive timely administration of antibiotics, emphasizing its poor sensitivity compared to contemporary clinical practice. Furthermore, the ESM was unable to identify 1,709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher (pre-selected threshold score) for 6,971 of all 38,455 hospitalized patients (18%), thus creating a large burden of alert fatigue for health care workers.
Image: PD
©2021 2 Minute Medicine, Inc. All rights reserved. No works may be reproduced without expressed written consent from 2 Minute Medicine, Inc. Inquire about licensing here. No article should be construed as medical advice and is not intended as such by the authors or by 2 Minute Medicine, Inc.
