Introduction: Accurate and timely diagnosis is crucial in the management of acutely and critically ill patients. Although large language models demonstrate exceptional performance in simulated scenarios, their diagnostic assistance efficacy in real-world clinical environments still requires rigorous clinical validation. This study evaluates the performance of the open-source large language model DeepSeek in assisting diagnostic decision-making upon hospital admission. Methods: We conducted a single-centre, retrospective evaluation involving 200 acutely and critically ill patients admitted to the Emergency Intensive Care Unit of Sun Yat-sen Memorial Hospital. Subsequently, three distinct methodologies were employed for independent assessment of admission diagnoses: (i) diagnosis by interns alone, (ii) diagnosis by DeepSeek alone, and (iii) diagnosis by interns assisted by DeepSeek. Final diagnoses established by a team of three chief physicians served as the clinical reference standard. Diagnostic accuracy, diagnostic scores, and time to diagnosis were compared among the three groups. Results: Interns alone achieved a 68.0% diagnostic accuracy, while DeepSeek alone correctly diagnosed 86.0% of cases. The intern-plus-DeepSeek diagnoses had 79.0% accuracy, outperforming interns alone and approaching DeepSeek’s performance. Diagnostic time was lowest with DeepSeek alone (2.1 ± 1.0 minutes), longest with interns alone (11.3 ± 4.5 minutes), and intermediate (7.0 ± 3.2 minutes) for the intern-plus-DeepSeek approach. DeepSeek assistance improved diagnostic accuracy among resident physicians by 11% (95% CI: 4.0–18.0%) and reduced time-to-diagnosis by 4.30 minutes on average (95% CI: 4.06–4.54%). The improvements in both accuracy and efficiency in the intern-plus-DeepSeek group were statistically significant (p < 0.001). Conclusions: DeepSeek demonstrates high diagnostic accuracy and effectively enhances the diagnostic performance of interns in acutely and critically ill patients. Although this preliminary study does not fully capture the iterative nature of real-world clinical care, our findings suggest that DeepSeek can serve as an accessible, open-source tool to assist clinicians, particularly in high-demand or resource-limited settings.
Huang et al. (Sun,) studied this question.