Evaluation of a ChatGPT-based artificial intelligence medical consultation system for frailty and sarcopenia related symptoms in older adults

ABSTRACT Background: With the rising prevalence of frailty and sarcopenia among aging populations, scalable and accessible consultation support tools are urgently needed. This study evaluates the performance of a ChatGPT-based Artificial Intelligence (AI) Medical Consultation System, developed using ChatGPT’s GPTs, in simulating structured clinical consultations, particularly for key frailty- and sarcopenia-associated symptoms. Objectives: To evaluate a ChatGPT-based AI Medical Consultation System’s ability to simulate structured clinical consultations for key frailty- and sarcopenia-related symptoms in older adults, and to characterize its performance in completeness, accuracy, redundancy, information categorization, and terminological consistency. Methods: The AI Medical Consultation System was designed with structured medical questions covering 390 clinical symptoms. Four chief complaints were selected from frailty and sarcopenia domains: gait disturbance, fatigue, muscle atrophy, and muscle weakness. The AI conducted consultations and generated structured medical histories. For each symptom, four iterations were conducted using detailed, patient-like prompts that reflect real-world geriatric clinical presentations. Performance was evaluated based on completeness, accuracy, redundancy, classification appropriateness, and terminological consistency. Results: The AI Medical Consultation System consistently produced structured summaries with appropriate classification across all four frailty and sarcopenia symptom domains. Mean word counts ranged from 291.25 to 352.5. However, frequent omissions were noted, such as family history, symptom details, and past medical history. Redundancy was observed in repeated content between the Medical History and Additional Information sections. Notably, the system at times prematurely summarized content or used inferred gendered pronouns without explicit input, highlighting unique AI-specific limitations. Conclusion: The AI Medical Consultation System shows promise as a structured data collection tool for frailty- and sarcopenia-related symptoms in older adults, offering detailed and largely accurate histories. However, it requires refinement in contextual understanding and content organization, particularly regarding redundancy elimination and explicit patient data handling.