Enhancing stability and reliability in LLM-driven robotic manipulation through human skill demonstration and visual tracking

Large Language Models (LLMs) have not only propelled advancements in natural language processing but also found increasingly extensive applications in robotics. However, due to insufficient integration of LLM-driven robotic systems in environmental perception, robotic skill operation, and language modeling, existing methods can only handle short-sequence tasks in simple and structured scenarios. We propose a novel LLM-driven robotic autonomous operation framework that leverages the Stable chain-of-thought and prompt engineering to optimize multi-step reasoning while employing behavioral self-correction to decompose high-level natural language instructions into robot action sequences based on human-defined preferences. Furthermore, by incorporating Beta Process Autoregressive Hidden Markov Model skill segmentation, along with Dynamic Movement Primitives, our framework achieves human–robot skill transfer, generalization, and sub-skill reuse, enabling rapid learning of anthropomorphic operations from human demonstrations. We also introduce a visual foundation model to provide comprehensive environmental perception and feedback, thereby enhancing adaptive generalization and adjustment capabilities. Extensive validation through simulations and real-world experiments confirms the efficacy of both individual components and the integrated system. Results demonstrate that our framework not only significantly outperforms baselines in reasoning stability, decision accuracy, and adaptability but also accomplishes complex trajectory tasks and skill generalization. Code and video are available at https://github.com/serafly/Robot-task-reasoning-and-skill-learning .