Non‐Destructive Freshness Assessment of Oysters Using a Multimodal Deep Learning Approach: Visual and Acoustic Fusion

ABSTRACT Oyster freshness plays a crucial role in determining food quality and safety. To enable rapid, accurate, and non‐destructive assessment, this study presents a multimodal deep‐learning framework that fuses visual and acoustic information for classifying oyster freshness. A custom‐designed acquisition system captures both top‐ and side‐view RGB images and vibration‐induced acoustic signals, which are processed through parallel multimodal branches for model processing. Using Total Volatile Basic Nitrogen (TVB‐N) as the reference index, 1050 oysters were classified into three freshness categories: Fresh, Sub‐fresh, and Spoiled. The proposed model demonstrates superior performance compared to unimodal approaches, achieving high accuracy and robustness for real‐time freshness monitoring. The network architecture comprises parallel modality‐specific feature extractors followed by an attention‐augmented fusion module that aligns and integrates cross‐modal semantic representations. To enhance generalization and mitigate overfitting, we employed CutMix data augmentation, Focal Loss to address class imbalance, and an early‐stopping protocol governed by validation accuracy. Empirical evaluations reveal that the proposed fusion model markedly surpasses unimodal counterparts, attaining an overall accuracy of 95.24%, an F1‐score of 0.9534, and an area under the ROC curve (AUC) of 0.9871. These results underscore the complementary nature of visual and acoustic modalities and highlight the potential of the presented approach for real‐time, scalable, and non‐invasive freshness monitoring of oysters and other aquatic products.