LLM4ATS: Applying Large Language Models for Auto-Testing Scripts in Automobiles

This paper introduces LLM4ATS, a framework integrating large language models, RAG, and closed-loop verification to automatically generate highly reliable automotive automated test scripts from natural language descriptions. Addressing the complex linguistic structure, strict rules, and strong dependency on the in-vehicle communication database inherent in ATS scripts, LLM4ATS innovatively employs fine-grained line-level generation and a rule-guided iterative refinement mechanism. The framework first enhances prompt context by retrieving relevant information from constructed syntax and case knowledge bases via RAG. Subsequently, each generated script line undergoes rigorous verification through a two-stage validator: initial syntax validation followed by semantic compliance checks against the communication database for signal paths and value domains. Any errors trigger structured feedback, driving iterative refinement by the large language model until fully compliant scripts are produced. This paper evaluated the framework’s effectiveness on real ATS datasets, testing models including GPT-3.5, GPT-4, Qwen2.5-7B, and Qwen2.5-72B-Instruct. Experimental results demonstrate that compared to zero-shot and few-shot baseline methods, the LLM4ATS framework significantly improves generation quality and pass rates across all models. Notably, the strongest GPT-4 model achieved a script pass rate of 91% with LLM4ATS, up from 42% in zero-shot mode, and validated functional effectiveness on a specified in-vehicle hardware platform (Chery Fengyun T28 dashboard). At the same time, expert manual evaluations confirmed the superior performance of the generated scripts in correctness, readability, and compliance with industry standards.