Memory length and space shape multi-agent Q-learning dynamics

In repeated interactions, players adjust their behavior based on previous moves. Higher memory leads to an exponential growth in the number of strategies, meaning players require more complex cognitive abilities. Even if players can observe the strategies and payoffs of their co-players, strategies imitated through social learning fail to guarantee effective responses to co-players to bring higher payoffs. We depict the human learning process through reinforcement learning, whereby players learn based on past experiences, independently of the strategies and payoffs of co-players. Here, we explore how different memory lengths and spaces, namely, memory-n, reactive-n, and reactive-n counting, affect the evolution of cooperation among reinforcement learning players. We found that memory-n players maintained higher cooperation than reactive-n players. Notably, higher memory promotes cooperation in memory-n players but inhibits it in reactive-n players. Reactive-n counting players can alleviate the negative effects of excessive memory by compressing the memory. Strategies with the nature of mutual cooperation and retaliation are key for reinforcement learning players to maintain cooperation. Our research highlights that judiciously adjusting the information available to players more effectively fosters cooperation within multi-agent systems.