LLM-guided expert feature extraction with fusion-based complementary learning network for fish freshness classification using eye images

Accurate, non-invasive grading of fish freshness remains challenging in dense and occluded environments, where the fish eye images serve as a reliable and consistently visible indicator of freshness. Loss of structural cues in the fish eye image often limits the existing deep learning models (DLN) from having consistent learning. To address these limitations, this work aims to develop a fusion-based complementary learning network (FCLN) that integrates expert-suggested photometric features with deep visual representations of DLN for the classification of fish freshness. Fish expert knowledge on freshness evaluation is processed using a large language model (LLM) to identify relevant visual cues and derive interpretable numeric features, eliminating the need for manual feature engineering. FCLN is constructed using ResNet50 as a backbone with a numeric feature layer and a fusion layer responsible for integrating both modalities for complementary learning. Complementary learning analysis (CLA) is also formulated to validate the complementary nature of the numeric features and image embeddings of DLNs. The proposed method is evaluated using the Fish Freshness Eye (FFE) dataset, which contains multiple fish species and freshness categories. Experimental results demonstrate a reliable classification accuracy of 81.03% during testing. CLA provides the least average correlation of 0.3755, indicating the complementary nature of the numeric features. Results of ablation studies also highlight the importance of using expert knowledge with deep visual features for fish freshness classification.