Artificial intelligence prediction model demonstrates limited potential for early sepsis detection

1. Kamran and colleagues used a retrospective inpatient cohort to evaluate the Epic sepsis model (ESM)’s ability to predict sepsis in earlier clinical stages.

2. ESM predictions made before the sepsis criteria were met demonstrated superior accuracy to random guesses. Its performance decreased when only analyzing predictions from earlier time points.

Evidence Rating Level: 2 (Good)

Study Rundown: Sepsis is a significant contributor to inpatient deaths, and early identification and treatment are crucial for improving mortality rates. The Epic sepsis model (ESM) is a widely used artificial intelligence (AI) predictive model that flags high-risk patients before sepsis onset. Kamran and colleagues conducted a study to evaluate the detection accuracy of ESM throughout hospitalization and relative to the time of sepsis treatment. The study included 77,582 eligible patients, and ESM used their electronic health records (EHR) to make predictions at 20-minute intervals throughout hospitalization. ESM’s predictive performance was evaluated before the sepsis criteria were met and before the first treatment indicator, and the performance was quantified as the area under the receiver-operating characteristic curve (AUROC). The study found that ESM predictions before the sepsis criteria were met achieved an AUROC of 0.62, while predictions before the first treatment indicators only had an AUROC of 0.47. Overall, this study demonstrated that ESM could reasonably predict sepsis before formal clinical diagnosis but has limitations in making accurate predictions at earlier time points.

Click here to read the study in NEJM AI

Relevant Reading: Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare

In-Depth [retrospective cohort]: The study used a retrospective inpatient cohort of 77,582 eligible patients, of which 3,766 (4.9%) developed sepsis. The researchers defined sepsis using the clinical definitions from the Centers for Disease Control and Prevention or the United States Centers for Medicare and Medicaid Services. The primary outcomes were ESM’s predictive performance before meeting the sepsis criteria and before the first treatment indicators. Sepsis predictions made throughout hospitalization were used as the upper bound on performance. Additional time points included predictions before ordering blood cultures, fluids, and treatments. ESM predictions before the sepsis criteria were met achieved an AUROC of 0.62 (95% CI, 0.61-0.63) compared to the upper bound of 0.87 (95% CI, 0.86-0.87). Performance decreased to 0.47 (95% CI, 0.46-0.48) when only using predictions before the first treatment indicators, where an AUROC of 0.5 meant that a model’s predictive performance was no better than random guesses. Performance dropped the most when predictions were limited to before ordering blood cultures. In 84.8% of the sepsis cases, treatment indicators occurred before the sepsis criteria were met, and the lack of available data could explain the lower performance. The study was limited by a lack of a universally accepted definition of sepsis and sample restriction to a single medical center. The authors concluded that the study had shown significant implications for developing and evaluating clinical sepsis prediction models and highlighted important AI design considerations for developers and researchers.

Image: PD

©2024 2 Minute Medicine, Inc. All rights reserved. No works may be reproduced without expressed written consent from 2 Minute Medicine, Inc. Inquire about licensing here. No article should be construed as medical advice and is not intended as such by the authors or by 2 Minute Medicine, Inc.