Scaling Up Multi-Agent Reinforcement Learning for Large Agent Teams and Long-Horizon Tasks: A Survey

Multi-agent reinforcement learning (MARL) empowers multiple autonomous agents to acquire effective policies for collaborative problem-solving. Over the last decade, MARL has seen significant advancements, with numerous algorithms achieving impressive performance across various benchmarks and real-world applications. Nevertheless, the scalability of multi-agent systems, in terms of the number of agents and the length of the task horizon, remains a critical consideration for applying MARL methods to complex problem-solving. Given that a dedicated review of the existing approaches and challenges in scaling up multi-agent systems remains largely absent, this survey aims to bridge this gap by delivering a comprehensive review of MARL methods developed to tackle challenging, scaled-up tasks. To this end, a novel taxonomy of MARL studies is introduced, categorizing them based on the external organizational control structures over all agents and the internal policy structures of individual agents. The survey also discusses the scales of popular MARL environments and tasks, providing a snapshot of the current challenging problems of interest. Furthermore, this survey underscores a set of critical open problems that call for further investigation in the field of scalable MARL.