Robust Multi-Object Fish Tracking in Dynamic Aquatic Environments via Attention-Enhanced YOLOv8 and LSTM-Based Trajectory Prediction

With the increasing refinement of ornamental fish culture, understanding fish behavioral patterns has become critical. Fish movements not only reflect daily activity ranges but also reveal responses to environmental changes such as water currents and obstacles. However, traditional manual observation is limited by manpower and time, making it difficult to record fish behaviors over long periods stably. Existing automated tracking techniques often suffer from ID switches and track interruptions caused by rapid fish movement, occlusions, or intermingling, which in turn degrade the reliability of subsequent analyses. This paper proposes a deep learning-based multi-object fish tracking system that integrates YOLOv8n for object detection and employs an IoU matching criterion to associate detections across consecutive frames, thereby maintaining object ID continuity. To further reduce ID loss under rapid motion and partial occlusion, a multiple-LSTM prediction model is introduced as a temporal compensation mechanism, thereby improving timing stability and track continuity. Moreover, considering disturbances in the experimental field (e.g., water disturbance and water current interference) that can blur fish body edges and fine details, an attention-enhanced detector, YOLOv8-CS (Convolutional Block Attention Module and Squeeze-and-Excitation), is developed by embedding CBAM and SE modules into the YOLOv8 architecture to enhance detection accuracy in dynamic waters. Experimental results demonstrate that the proposed system effectively increases the Multiple Object Tracking Accuracy (MOTA) to 77.23% and significantly reduces ID switches to 42.5, ensuring more robust and continuous trajectory tracking compared to benchmark methods. This system provides a highly reliable tool for automated behavior analysis in complex and dynamic aquatic environments.