Abstract The actions of animals provide a window into how their minds work. Recent advances in deep learning are providing powerful approaches to recognize patterns of animal movement from video recordings using markerless pose estimation models. Current methods for classifying animal behaviour using the outputs of these models often rely on species and task-specific feature engineering of trajectories, kinematics and task programming. Generalized solutions that use only pose estimations and the inherent structure of animals and their environment provide an opportunity to develop foundational, contextual and, importantly, standardized animal behaviour models for efficient and reproducible behavioural analysis. Here, we present PoseRecognition (PoseR), a behavioural classifier using spatio-temporal graph convolutional networks. We show that it can be used to classify animal behaviour quickly and accurately from pose estimations, using zebrafish larvae, Drosophila melanogaster, mice and rats as model organisms. Our easily accessible tool simplifies the behavioural analysis workflow by transforming coordinates of animal position and pose into semantic labels with speed and precision. The design of our tool ensures scalability and versatility for use across multiple species and contexts, improving the efficiency of behavioural analysis across fields.
Mullen et al. (Wed,) studied this question.