Artificial intelligence-augmented decision-making failed to show non-inferiority compared to clinician judgment alone for coronary angiography patient selection

1. Upton and colleagues assessed the appropriateness of coronary angiography referrals made by clinicians working independently or with an AI tool.

2. The primary outcome difference between the two groups did not meet the non-inferiority margin, and other metrics did not show significant differences.

Evidence Rating Level: 1 (Excellent)

Study Rundown: Clinicians rely on non-invasive imaging, such as stress echocardiography, to assess patients with suspected cardiac diseases. However, echocardiography interpretation relies on qualitative assessments. Upton and colleagues conducted a randomized controlled trial (RCT) that examined AI’s ability to augment clinician decision-making. Eligible patients with suspected coronary artery disease (CAD) were blinded and randomly assigned in a 1:1 ratio to the standard decision-making (control) group or the AI-augmentation (intervention) group. In the control group, patients proceeded down their care pathways based on clinician decisions. In the intervention group, echocardiography images were also assessed by the AI-augmentation device, which informed the clinician whether the patient had a low or high probability of severe CAD. The primary outcome was the appropriateness of coronary angiography referral assessed by the area under the receiver operating characteristic curve (AUROC), as well as sensitivity and specificity. The study found that the AUROCs of the two groups failed to meet the prespecified non-inferiority margin; the two groups had no statistically significant differences between their sensitivity and specificity. This study demonstrated AI’s limitation in augmenting stress echocardiography interpretation in large volumes.

Click here to read the study in NEJM AI

Relevant Reading: Blinded, randomized trial of sonographer versus AI cardiac function assessment

In-Depth [randomized controlled trial]: Upton and colleagues enrolled 4,907 patients from 20 stress echocardiogram centers. Eligible patients were adults referred to ischemic heart disease investigation without significant comorbidities, serious cardiac diseases, or past cardiac surgery. All patients were followed for 6 months after the echocardiogram. An adjudication committee determined the appropriateness of the coronary angiography referral. True positives were defined as severe CAD requiring revascularization among referred patients, and false negatives were acute coronary events within 6 months. Patients were blinded to their group allocation, and the adjudication committee was blinded to the stress echocardiography results and the AI report. In the end, 2,213 (94.53%) patients completed the follow-up. The control group had an AUROC of 0.55 (95% CI, 0.33 to 0.80), while the intervention group had an AUROC of 0.63 (95% CI, 0.43 to 0.83. The difference between the two groups was 0.09 (95% CI, -0.22 to 0.39), which did not meet the pre-specified non-inferiority margin of -0.05. Their sensitivity and specificity also did not differ significantly, with a difference of 9.05 (95% CI, -21.8 to 39.59) for the former and -0.58 (95% CI, -2.13 to 0.94) for the latter. However, a subgroup analysis found a significant AUROC difference of 0.25 (95% CI, -0.02 to 0.62) in low-volume stress echocardiography centers. The authors concluded that AI augmentation did not show significant differences compared to standard decision-making but may benefit decisions made in low-volume centers.

Image: PD

©2024 2 Minute Medicine, Inc. All rights reserved. No works may be reproduced without expressed written consent from 2 Minute Medicine, Inc. Inquire about licensing here. No article should be construed as medical advice and is not intended as such by the authors or by 2 Minute Medicine, Inc.