ABSTRACT To enable autonomous operations in complex industrial environments, this paper proposes retrieval‐augmented generation with large language models for robotic operations (RAGLRO), a robotic framework specifically designed for power switchgear operation tasks. The system integrates multimodal perception with high‐level semantic reasoning and task‐level action generation. A depth camera captures the environmental context, which is processed by visual modules to perform object detection and pose detection. The perception outputs are formulated into structured prompts and provided to a large language model (LLM) equipped with a retrieval‐augmented generation (RAG) mechanism. The RAG component enables the LLM to dynamically access a task‐specific knowledge base, including operation manuals, safety protocols and historical mission data, thereby enhancing contextual understanding and reasoning precision. Based on the retrieved knowledge and current environmental perception, the LLM selects and sequences callable action functions from a predefined robotic action library to generate executable robot control commands. A dedicated dataset for power switchgear operations is also constructed to support robust visual perception, containing annotated images for object detection and pose detection tasks. Experimental results demonstrate that RAGLRO achieves high task success rates and strong adaptability in real‐world power maintenance scenarios, validating the effectiveness of integrating multimodal perception, LLM‐based reasoning and RAG‐grounded task planning within a unified robotic control framework.
Wang et al. (Fri,) studied this question.