Healthcare agent: eliciting the power of large language models for medical consultation

Abstract Large language models (LLMs) have achieved remarkable success across many fields but face challenges in complex real-world scenarios like medical consultation, particularly regarding inquiry quality and safety concerns. In this paper, we introduce a healthcare agent designed to address these issues, which cannot be fully resolved through a vanilla one-time fine-tuning process. The healthcare agent includes three components: a dialogue component for planning safe and effective conversations, a memory component for storing patient conversations and medical history, and a processing component for report generation. To evaluate our healthcare agent’s performance in medical consultations, we employ both expert assessment from medical professionals and an automated evaluation system powered by ChatGPT for large-scale testing. Our results demonstrate that the healthcare agent significantly enhances the capabilities of general LLMs in medical consultations, particularly in inquiry quality, response quality, and safety. Through extensive ablation studies, we also analyze the impact of each component.