• About
  • Masthead
  • License Content
  • Advertise
  • Press Releases
  • Newsroom
  • RSS/Email List
  • 2MM Podcast
  • Write for us
  • Contact Us
2 Minute Medicine
No Result
View All Result

No products in the cart.

Sign in
SUBSCRIBE
  • About
  • Specialties
    • All Specialties, All Recent Reports
    • Cardiology
    • Chronic Disease
    • Dermatology
    • Emergency
    • Endocrinology
    • Gastroenterology
    • Imaging and Intervention
    • Infectious Disease
    • Nephrology
    • Neurology
    • Obstetrics
    • Oncology
    • Ophthalmology
    • Pediatrics
    • Pharma
    • Preclinical
    • Psychiatry
    • Public Health
    • Pulmonology
    • Rheumatology
    • Surgery
    • Rewinds
    • The Scan
  • EvidencePulse™
  • HypeCheck™
  • AI News
  • Classics™
    • 2MM+ Online Access
    • Paperback and Ebook
  • Tools
    • NPI Registry Lookup
    • RVU Search
    • Physician Pay by Specialty
    • ICD-10 Code Lookup
    • FDA Drug Approvals
    • Drug Safety for Breastfeeding/Pregnancy
    • Clinical Guidelines
    • Medical Calculators
    • ECG Reference
      • QTc Calculator
      • Sgarbossa Criteria
      • ECG Axis Calculator
      • AV Block Helper
  • Rankings
    • Medical School Rankings
    • Residency Rankings
  • Partners
    • License Content
    • Advertise with Us
2 Minute Medicine
  • About
  • Specialties
    • All Specialties, All Recent Reports
    • Cardiology
    • Chronic Disease
    • Dermatology
    • Emergency
    • Endocrinology
    • Gastroenterology
    • Imaging and Intervention
    • Infectious Disease
    • Nephrology
    • Neurology
    • Obstetrics
    • Oncology
    • Ophthalmology
    • Pediatrics
    • Pharma
    • Preclinical
    • Psychiatry
    • Public Health
    • Pulmonology
    • Rheumatology
    • Surgery
    • Rewinds
    • The Scan
  • EvidencePulse™
  • HypeCheck™
  • AI News
  • Classics™
    • 2MM+ Online Access
    • Paperback and Ebook
  • Tools
    • NPI Registry Lookup
    • RVU Search
    • Physician Pay by Specialty
    • ICD-10 Code Lookup
    • FDA Drug Approvals
    • Drug Safety for Breastfeeding/Pregnancy
    • Clinical Guidelines
    • Medical Calculators
    • ECG Reference
      • QTc Calculator
      • Sgarbossa Criteria
      • ECG Axis Calculator
      • AV Block Helper
  • Rankings
    • Medical School Rankings
    • Residency Rankings
  • Partners
    • License Content
    • Advertise with Us
Sign in
SUBSCRIBE
2 Minute Medicine
Sign in
Subscribe
2 Minute MedicineFree on Google PlayGet
✦EvidencePulse™by 2 Minute Medicine3 reportsYou askedtop stroke trials 2026Synthesizing medical evidence...Top 2026 stroke trial results:OCEANIC — ischemic stroke: 6.2% vs 8.4%OPTION* — mRS 0–1: 43.6% vs 34.2%ORIENTAL* — mRS 0–2: 58.6% vs 46.6%*Higher sICH in intervention armsParticipants randomizedNOCEANIC12,327OPTION570ORIENTAL564Ask about guidelines or landmark trials...↑Try EvidencePulse™Ask the evidence. Get the signal.Try it now →
Home 2 Minute Medicine

Large language models (LLMs) demonstrated higher error rates compared to humans in a clinical oncology question bank

byCheng En Xi, BHScandDeepti Shroff, DMD, MS
May 28, 2024
in 2 Minute Medicine, Oncology
Reading Time: 3 mins read
Share on FacebookShare on TwitterShare on LinkedIn

1. A comparative evaluation tested five publicly available LLMs on 2044 oncology questions, covering comprehensive topics in the field. The responses were compared to a human benchmark.

2. Only one of the five models tested performed above the 50th percentile, with worse performance observed in clinical oncology subcategories and female-predominant malignancies.

Evidence Rating Level: 2 (Good)

Study Rundown: Many medical professionals have begun to use large language models (LLMs), such as ChatGPT, as augmented search engines for medical information. LLMs have demonstrated high performance on subspecialty medical examinations across multiple medical specialties, but the utility of LLMs in clinical applications of clinical oncology remains unexplored.

Rydzewski and colleagues compared the performance of five LLMs on a set of multiple-choice questions related to clinical oncology to a random guess algorithm and the performance of radiation oncology trainees. The authors assessed the accuracy of the models, their self-appraised confidence, and consistency of responses across three independent replicates of questions. The LLMs were asked to provide an answer to a question, a confidence score, and an explanation of the response. Each LLM was evaluated with 2044 unique questions, across three independent replicates.

The study found that only one of the five LLMs (GPT-4) scored higher than the 50th percentile when compared to human trainees, despite all showing high self-appraised confidence. The remaining LLMs had much lower accuracies, with some being similar to the random guess strategy. LLMs scored higher on foundational topics and worse on clinical oncology topics, especially ones related to female-predominant malignancies. The authors found combining model selection, self-appraised confidence, and output consistency, helped identify more reliable outputs. Overall, this study demonstrated a need to assess the safety of implementing LLMs in clinical settings and the presence of training bias, in the form of medical misinformation related to female-predominant malignancies.

RELATED REPORTS

AI scribe use associated with reduced documentation time but longer clinical notes

An artificial intelligence system for human papillomavirus-positive dual stain cervical cancer screening reads p16-Ki67 slides

Click here to read the study in NEJM AI

Click to read an accompanying editorial in NEJM AI

Relevant Reading: Performance of ChatGPT on a primary FRCA multiple choice question bank

In-Depth [cross-sectional study]: In this study, Rydzewski and colleagues assessed the accuracies of five LLMs (LLaMA, PaLM 2, Claude-v1, GPT-3.5, and GPT-4) in 2044 multiple choice questions and aimed to identify strategies to help end users identify reliable LLM outputs. The questions were sourced from the American College of Radiology in-training radiation oncology examinations from 2013-2017, 2020, and 2021. Each question was repeated across three independent replicates. The authors compared LLM performance with a random guessing strategy and human scores for the questions sourced from the 2013 and 2014 examinations. Also, the authors assessed the self-appraised confidence of the LLMs by prompting for a confidence score ranging from 1-4, with 1 indicating a random guess and 4 indicating maximal confidence.

The five LLMs had mean accuracies ranging from 25.6%-68.7%, as compared to 25.2% of the random guess strategy. When compared against humans, only GPT-4 scored higher than 50th percentile, achieving 69th and 89th percentiles. The overall performances of LLMs were positively correlated (Pearson’s r = 0.630; p < 0.001) with their performances in a specific topic. Other than LLaMA 65B, all LLMs performed better on foundational topics (e.g., medical statistics, cancer biology), than clinical subcategories (p < 0.02). LLMs performed the worst with regards to subjects involving breast and gynecologic malignancies. All LLMs produced a confidence score of 3 or 4 in more than 94% of responses. Finally, by combining self-assessed confidence and output consistency, the authors generated accuracies of 81.7% and 81.1% in Claude-v1 and GPT-4, respectively.

In conclusion, the authors assessed the ability of five LLMs to answer clinical oncology examination questions. This work displayed the need for further LLM safety evaluations before routine clinical implementation. It also provided insight into a potential strategy to more reliably use LLM output.

Image: PD

©2024 2 Minute Medicine, Inc. All rights reserved. No works may be reproduced without expressed written consent from 2 Minute Medicine, Inc. Inquire about licensing here. No article should be construed as medical advice and is not intended as such by the authors or by 2 Minute Medicine, Inc.

Tags: artificial intelligencelarge language modelsoncology
Previous Post

Roche: CT-388 Dual GLP1-GIP Receptor Agonist Clinically Improves Weight Loss

Next Post

The Scan by 2 Minute Medicine®: Keeping up with Keto, Feeding with HIV, Lung Cancer Overhaul and Catastrophic Chip Challenge

RelatedReports

AI scribes collage: a wide vintage microphone under three blank speech bubbles, with an empty clinic chair and exam table
Artificial Intelligence

AI scribe use associated with reduced documentation time but longer clinical notes

November 6, 2026
A calendar, a specimen jar, and a brush are depicted alongside a microscopic view of cervical cells, indicating cervical cancer screening.
All Specialties

An artificial intelligence system for human papillomavirus-positive dual stain cervical cancer screening reads p16-Ki67 slides

October 23, 2026
An illustration featuring a heart within a house, with a stethoscope, a scale, and medication, symbolizing heart failure home hospital care.
Artificial Intelligence

An artificial intelligence model demonstrates the potential for improving the efficiency of structural heart disease screening using electrocardiograms

October 21, 2026
A sleeping child with electrodes on the scalp, surrounded by a moon and night sky, illustrating sleep monitoring and brain health assessment.
Artificial Intelligence

Deep learning model demonstrates potential to generate an interpretable brain health biomarker from sleep electroencephalography

October 19, 2026
Next Post
The Scan by 2 Minute Medicine®:  Ultra-Trail du Mont-Blanc, Taylor Swift, NBA rookie Chet Holmgren and Magic Mushrooms!

The Scan by 2 Minute Medicine®: Keeping up with Keto, Feeding with HIV, Lung Cancer Overhaul and Catastrophic Chip Challenge

#VisualAbstract: Intensive Ambulance-Delivered Blood-Pressure Reduction Does Not Improve Outcomes in Hyperacute Stroke

#VisualAbstract: Intensive Ambulance-Delivered Blood-Pressure Reduction Does Not Improve Outcomes in Hyperacute Stroke

Mental wellbeing collage: a meditation cushion by an open window with lifting curtains, a potted fern and a folded shawl

Bayer AG: Elinzanetant Shows Promise In Treating Menopause-Associated Vasomotor Symptoms In Phase 3 Clinical Trials

✦ EvidencePulse™ by 2 Minute Medicine 3 reports You asked top stroke trials 2026 Synthesizing medical evidence... Top 2026 stroke trial results: OCEANIC — ischemic stroke: 6.2% vs 8.4% OPTION* — mRS 0–1: 43.6% vs 34.2% ORIENTAL* — mRS 0–2: 58.6% vs 46.6% *Higher sICH in intervention arms Participants randomizedN OCEANIC 12,327 OPTION 570 ORIENTAL 564 Ask about guidelines or landmark trials... Try EvidencePulse™ Get the signal. Try it now → Physician-trained medical AI by 2 Minute Medicine

2MM+ All Access

Full details →
$4.99 /month

3-day free trial · cancel anytime

  • ✓100+ physician-written reports
  • ✓EvidencePulse™ + HypeCheck™
  • ✓Ad-free reading + newsletters
Start 3-day free trial

Then $4.99/month. Manage renewal in your account.

2 Minute Medicine® is an award winning, physician-run, expert medical media company. Our content is curated, written and edited by practicing health professionals who have clinical and scientific expertise in their field of reporting. Our editorial management team is comprised of highly-trained MD physicians. Join numerous brands, companies, and hospitals who trust our licensed content.

Recent Reports

  • Cretostimogene reduced recurrence risk in intermediate-risk and high-grade Ta non-muscle invasive bladder cancer in topline phase 3 results
  • Removing Race and Ethnicity From Dialysis Mortality Risk Adjustment Changes Facility Performance Rankings
  • AI Model Shows Promise in Predicting PTSD Risk Following Trauma
License Content
Terms of Use | Disclaimer
Cookie Policy

Get the app

Read 2 Minute Medicine on iPhone, iPad and Android.

Download on the App StoreGet it on Google Play
QR code for www.2minutemedicine.com/app
On a computer? Scan this with your phone’s camera.
2MM+ All Access Full details →

The medical literature, distilled daily.

Monthly plan selected: $4.99 /month.

$4.99/month

3-day free trial · cancel anytime

Start 3-day free trial

Then $4.99/month. Manage renewal in your account.

Yearly plan selected: $49.90 /year.

$49.90/year

2 months free

Start 3-day free trial

Then $49.90/year.

Membership benefits

  • ✓100+ new physician-written study summaries monthly, 12,000+ searchable
  • ✓AI EvidencePulse™ + HypeCheck™
  • ✓Ad-free reading + newsletters
  • ✓FDA drug approvals + clinical calculators

Already a member? Sign in

  • Get the app
  • About
  • Specialties
    • All Specialties, All Recent Reports
    • Cardiology
    • Chronic Disease
    • Dermatology
    • Emergency
    • Endocrinology
    • Gastroenterology
    • Imaging and Intervention
    • Infectious Disease
    • Nephrology
    • Neurology
    • Obstetrics
    • Oncology
    • Ophthalmology
    • Pediatrics
    • Pharma
    • Preclinical
    • Psychiatry
    • Public Health
    • Pulmonology
    • Rheumatology
    • Surgery
    • Rewinds
    • The Scan
  • EvidencePulse™
  • HypeCheck™
  • AI News
  • Classics™
    • 2MM+ Online Access
    • Paperback and Ebook
  • Tools
    • NPI Registry Lookup
    • RVU Search
    • Physician Pay by Specialty
    • ICD-10 Code Lookup
    • FDA Drug Approvals
    • Drug Safety for Breastfeeding/Pregnancy
    • Clinical Guidelines
    • Medical Calculators
    • ECG Reference
      • QTc Calculator
      • Sgarbossa Criteria
      • ECG Axis Calculator
      • AV Block Helper
  • Rankings
    • Medical School Rankings
    • Residency Rankings
  • Partners
    • License Content
    • Advertise with Us
No Result
View All Result

© 2026 2 Minute Medicine, Inc. - Physician-written medical news.