Data-driven simulator of multi-animal behavior with unknown dynamics via reinforcement learning

Advances in imitation learning for robotics have expanded the possibilities for imitating human and animal movements. However, simulating realistic multi-animal behaviors in biology remains challenging because transition models are often unknown. Since locomotion dynamics are seldom known, one cannot rely solely on mathematical models, and constructing a simulator that reproduces trajectories and supports reward-driven optimization remains a research gap. Here we introduce a data-driven simulator of multi-animal behavior using deep reinforcement learning with (counterfactual) simulations. We address the ill-posed problem caused by high degrees of freedom via estimating movement variables in reinforcement learning. We also use a distance-based pseudo-reward to align and compare states between cyber and physical spaces. We verified our approach using data from artificial agents, flies, newts, and silkmoth, revealing higher reproducibility and reward acquisition than simple imitation learning and reinforcement learning approaches. Furthermore, our approach enables counterfactual behavior prediction in unknown experimental settings. These suggest the potential to simulate and understand complex multi-animal behaviors.