What question did this study set out to answer?

This study aims to assess the effectiveness of large language models in accurately classifying bladder cancer lesions using the VI-RADS framework.

April 13, 2026BMC Medical Imaging1 citationsOpen Access

Evaluation of large language models for VI-RADS reports: a comparative analysis of zero-shot and few-shot prompting

AHAhmet Halis DÇDeniz Çeliker

Key Points

This study aims to assess the effectiveness of large language models in accurately classifying bladder cancer lesions using the VI-RADS framework.
Developed a synthetic dataset of 100 bladder cancer cases based on expert-crafted reports.
Evaluated three large language models: ChatGPT, Gemini, and Copilot under zero-shot and few-shot prompting conditions.
Measured performance using accuracy, macro F1 scores, and Cohen’s kappa with statistical significance assessed using McNemar's test.
Few-shot prompting significantly boosted classification accuracy for ChatGPT (94%) and Gemini (83%) compared to zero-shot prompting.
Copilot's accuracy declined to 53% under few-shot conditions, highlighting variability in model performance.
ChatGPT showed the highest consistency in identifying high-risk lesions (VI-RADS categories 4–5).

Abstract

Accurate preoperative staging is vital in bladder cancer management, particularly for assessing muscle invasion. Multiparametric MRI (mpMRI) combined with the Vesical Imaging–Reporting and Data System (VI-RADS) offers a non-invasive and standardized approach for tumor stratification. However, inter-observer variability among radiologists remains a significant limitation. This study investigates the performance of three large language models (LLMs)—ChatGPT (OpenAI; GPT-5.2), Gemini (Google; Gemini 2.0), and Copilot (Microsoft; Copilot based on GPT-5 architecture)—in classifying bladder lesions according to VI-RADS using zero-shot and few-shot prompting strategies. A synthetic dataset of 100 simulated bladder cancer cases was developed from expert-crafted synthetic radiology reports based on structured text descriptors and expert consensus, comprising 20 cases for each VI-RADS category (1–5). Each case included structured imaging descriptors aligned with VI-RADS scoring criteria. The LLMs were evaluated under zero-shot (no examples provided) and few-shot (with illustrative examples) prompting conditions. Performance metrics included accuracy, macro F1 scores, and Cohen’s kappa, with statistical significance assessed using McNemar’s test. Few-shot prompting significantly improved the classification performance of ChatGPT (OpenAI; GPT-5.2) (accuracy: 94%, F1: 0.939) and Gemini (Google; Gemini 2.0) (accuracy: 83%, F1: 0.823) compared to zero-shot. In contrast, Copilot (Microsoft; Copilot based on GPT-5 architecture)‘s accuracy (53%) and F1 score (0.503) declined under few-shot conditions. ChatGPT (OpenAI; GPT-5.2) demonstrated the highest consistency in identifying high-risk lesions (VI-RADS 4–5), followed by Gemini (Google; Gemini 2.0). Few-shot prompting enhances LLM performance in VI-RADS classification, particularly for ChatGPT (OpenAI; GPT-5.2) and Gemini (Google; Gemini 2.0). These findings highlight the potential of AI tools to support radiological decision-making in bladder cancer staging. Further studies using real imaging data and explainable AI are warranted.

问 AI

Bookmark

View Full Paper