Large language models are poor clinical administrators: An evaluation of structured queries in real-world electronic health records

Large language models (LLMs) have shown promise in interpreting clinical free-text like provider notes. There is limited evidence on tabular electronic health record (EHR) tasks. Our objective was to evaluate the accuracy of LLMs on structured EHR administrative tasks using direct prompting, chain-of-thought (CoT) reasoning, and tool-enabled code generation. We evaluated nine LLMs randomly sampling from a real-world sampled dataset of 50,000 emergency department (ED) visits. Tasks were tested across 25 combinations of table sizes (5–25 rows and columns). Models were prompted directly or with CoT reasoning to return numerical answers. In the tool setting, models generated Python code, which was executed to retrieve answers. Accuracy was defined as the proportion of model outputs matching validated references. We also assessed JSON format compliance. Across 32,950 model queries, performance varied by model, task type, and prompting strategy. Direct prompting produced uniformly low accuracies. CoT prompting moderately improved performance, particularly for logical filtering, but results degraded significantly as table size increased. The tool-based strategy substantially improved accuracy. Smaller models and distilled reasoning variants had more frequent formatting and execution errors. In conclusion, for structured EHR tabular data extraction, direct and CoT prompting strategies resulted in limited accuracy and poor scalability, particularly as table size increased. Tool-based prompting, where models generated and executed Python code, achieved higher accuracy and valid output formatting. Structured data tasks in clinical workflows may require hybrid approaches that combine LLMs with code execution to ensure accuracy and consistency.