Air combat joint strategy learning based on a dual-loop framework and hindsight experience replay

Abstract The research on air combat decision-making methods using artificial intelligence has become a widely studied field. However, due to the complexity of the air combat process and the problem of hybrid action selection (discrete/continuous), traditional methods struggle to simultaneously make decisions on continuous maneuvering and discrete missile launching actions. In addition, designing complex dense reward functions requires difficult-to-obtain aviation expert knowledge, while relying on sparse reward functions makes it difficult to fully explore a large state space. In view of this, we propose a novel algorithm based on a dual-loop framework. The core idea is to separate maneuvering and missile launching decisions into two optimization processes within the training loop, enabling joint decision-making during the search phase while allowing independent optimization during the optimization phase. Besides, hindsight experience replay is adopted to train missile launching decisions. It expands valuable learning samples through a sample relabeling approach. We designed a series of experiments to validate the performance of the proposed method by constructing the opponent’s strategy using a self-play agent and an air combat bot. The performance of the proposed method was validated in a simulation environment, demonstrating that it can generate an air combat joint strategy incorporating both maneuvering and missile launching. In adversarial experiments, the air combat joint strategy we generated achieved a higher win rate than other state-of-the-art air combat methods.