What is the clinical evidence from this study?

Study design: Other. Population: Perioperative medication discontinuation (n=105). Intervention: Generative AI models (OpenAI o3, Gemini 2.5 Pro Exp, GPT-4o, etc.) vs. Clinical pharmacist consensus (reference standard). Primary outcome: Accuracy of perioperative medication discontinuation decisions.

What question did this study set out to answer?

The aim is to assess the performance of various generative AI models in making perioperative medication discontinuation decisions.

April 12, 2026Iryo Yakugaku (Japanese Journal of Pharmaceutical Health Care and Sciences)0 citationsOpen Access

Comparison of the Performance of Generative Artificial Intelligence Models in Perioperative Medication Discontinuation: A Clinical Pharmacy Evaluation

DMDaiki MakiedaOkayama University Hospital HIHiromi IdaOkayama University Hospital SHSatoko HiroeOkayama University Hospital

Key Result

OpenAI o3 demonstrated the highest accuracy (87.6%) among nine generative AI models for perioperative medication discontinuation decisions, followed by Gemini 2.5 Pro Exp (84.8%) and GPT-4o (83.8%).

Key Points

The aim is to assess the performance of various generative AI models in making perioperative medication discontinuation decisions.
Evaluated nine generative AI models available in April 2025.
Utilized 15 mock prescription sets with 105 items each.
Each model received a standardized prompt assessed by five pharmacists with clinical experience.
Outputs were evaluated based on drug identification, discontinuation timing, and pharmacological rationale.
OpenAI o3 achieved the highest accuracy at 87.6%.
Gemini 2.5 Pro Exp followed with 84.8% accuracy, while GPT-4o had 83.8%.
Lightweight models performed poorly, especially on OTC products and dietary supplements.
Top-performing models could support decision-making, but 10-20% incorrect responses were noted.

Structured PICO

How accurate are various generative AI models in determining perioperative medication discontinuation?

Population

15 mock prescription sets comprising 105 items (including OTC drugs and dietary supplements) evaluated by 5 hospital pharmacists with ≥5 years of clinical experience.

Intervention

Evaluation by nine generative AI models (GPT-4o, GPT-4o mini, OpenAI o3, Gemini 2.5 Pro Exp, Gemini 2.0 Flash, Claude 3.7 Sonnet, Grok 3, Llama 4 Scout, and DeepSeek R1) using a standardized Japanese prompt.

Comparator

Comparison among the different AI models.

Outcome

Accuracy of responses based on accurate drug identification, appropriateness of discontinuation and resumption timing, and validity of the pharmacological rationale (considered correct when at least 4 of 5 pharmacists agreed).

Advanced generative AI models can accurately support perioperative medication discontinuation decisions in approximately 85% of cases, but still require pharmacist verification due to a 10-20% error rate.

Limitations

Single-center study with a limited number of mock prescription sets (15 sets, 105 items), which may depend on institutional characteristics.
Verification was conducted under limited conditions using only prescription information without patient background data.
Results depend on the specific generative AI models and versions available at the time of evaluation, which may change with future updates.

Abstract

Generative artificial intelligence (AI) has rapidly advanced and is expected to enhance the efficiency and quality of clinical practice. In pharmacy practice, determining perioperative medication discontinuation is a critical task that is directly related to patient safety. However, standardized guidance remains limited, particularly for over-the-counter (OTC) drugs and dietary supplements, and decisions often rely on the individual expertise of pharmacists. In this study, the accuracy of nine generative AI models available in April 2025 (GPT-4o, GPT-4o mini, OpenAI o3, Gemini 2.5 Pro Exp, Gemini 2.0 Flash, Claude 3.7 Sonnet, Grok 3, Llama 4 Scout, and DeepSeek R1) for perioperative medication discontinuation decisions was evaluated using 15 mock prescription sets comprising 105 items. Each model received a standardized Japanese prompt, and outputs were independently assessed by five hospital pharmacists (≥5 years of clinical experience) based on three criteria: accurate drug identification, appropriateness of discontinuation and resumption timing, and the validity of the pharmacological rationale. A response was considered correct when at least four of the pharmacists agreed. OpenAI o3 demonstrated the highest accuracy (87.6％), followed by Gemini 2.5 Pro Exp (84.8％) and GPT-4o (83.8％). Lightweight models demonstrated lower accuracy, particularly for OTC products, dietary supplements, and fixed-dose combination drugs. High-performance models with advanced reasoning capabilities exhibited high accuracy and may serve as useful decision-support tools. However, incorrect responses occurred in approximately 10 – 20％ of cases, even among the top-performing models. Therefore, safe clinical implementation requires careful model selection, integration with institutional knowledge resources, and final verification by pharmacists.

AI에게 질문

Bookmark

View Full Paper