Machine-learning-based models found to have predictive abilities no better than chance in out-of-sample evaluations

1. Machine learning prediction models fared poorly when used across clinical trials

2. To improve generalizability, study authors highlighted the need for more detailed phenotyping and longitudinal validation within clinical trials predictive models

Evidence Rating Level: 1 (Excellent)

Study Rundown: The research landscape of predictive machine learning for health outcomes has expanded tremendously in the past few years. A central promise of these flexible modeling approaches is the ability to generalize findings to contexts outside of which they were developed. Indeed, sophisticated artificial intelligence methods have begun to emerge as veritable tools used in clinical trial analysis. Yet, a largely underexplored evaluation metric is how well these tools perform out-of-sample (i.e. across clinical trials).

Chekroud and colleagues used the Yale Open Data Access (YODA) Project to gather treatment data from five international, multi-site randomized control trials (RCTs) that were conducted to evaluate the efficacy of antipsychotic drugs for patients with schizophrenia. Surprisingly, the authors found that, for most trial-trial pairs and across nearly all clinical outcomes measured in these trials, the cross-validity of these prediction models showed no better accuracy than a coin flip.

The authors conclude by suggesting some reasons for a lack of generalizability and ways to improve machine learning predictability. One strategy would employ the use of longitudinal validation cohorts, wherein the validation of a machine learning framework could be performed at a later time point for the same trial participants. Overall, this study offers a cautionary tale for the machine learning field in how to best evaluate new methods for clinical prediction.

Click here to read the study in Science.

Relevant Reading: Computational approaches and machine learning for individual-level treatment predictions

In-Depth [meta-analysis]: Chekroud and colleagues evaluate the predictability of two commonly used machine learning prediction models, the elastic net regression and random forest algorithm, between clinical trials of antipsychotic efficacy for patients with schizophrenia. The authors used data from five multi-center RCTs to build prediction models for metrics related to the Positive and Negative Syndrome Scale for Schizophrenia (PANSS) and then evaluated sensitivity, specificity, and the area under the receiver operator curve (AUROC) between trials.

The authors first analyzed whether prediction models trained on one trial would exhibit any predictive capacity in another independent RCT. Save for a few trial pairs, nearly all prediction models trained on RCT showed a balanced accuracy (defined as the average of the sensitivity and specificity metric) estimate nearly equal to random chance (0.5). Indeed, this same result was replicated across a wide range of metrics based on the PANSS (i.e. 50% reduction or 25% reduction in PANSS, the percentage change in PANSS total score, etc.). These results were replicable across prediction model types – both the elastic net and random forest frameworks showed similar cross-trait poor predictability.

The authors conclude by emphasizing reasons for such poor generalizability, including inadequate data, highly contextualized trials, and diversity among patient populations. With these lessons in mind, they identify some areas for improving generalizability. One such option would be to identify trial-level factors influencing patient outcomes, thereby stratifying data to identify more meaningfully similar trials. In sum, this article offers a carefully constructed meta-analysis of clinical trial data to highlight the need for improved generalizability of machine learning predictive models across large trial datasets.

Image: PD

©2024 2 Minute Medicine, Inc. All rights reserved. No works may be reproduced without expressed written consent from 2 Minute Medicine, Inc. Inquire about licensing here. No article should be construed as medical advice and is not intended as such by the authors or by 2 Minute Medicine, Inc.