Cognition-driven Adaptive Semantic Decoding Framework for Multimodal Sentiment Analysis

In real-world scenarios, multimodal sentiment analysis faces significant challenges, particularly in cross-scenario generalization. Existing works fails to effectively deal with the variability in evaluation frameworks and modality combinations, which results in the poor transfer performance across different application contexts. In this paper, the cognition-driven adaptive semantic decoding framework (CASDF) is proposed to realize evaluation system and modality independent multimodal sentiment analysis. Specifically, the adaptive modality association module is proposed to construct the adaptive modality mapping space, which allows us to dynamically adapt to arbitrary modality combinations. This indeed breaks through the limitation of the modality number and effectively deal with the modality gap. Furthermore, similar to the human hierarchical cognition (”perception-concept-decision”), the evaluation system progressive alignment module is presented to establish the unified evaluation system. This is consists of the perception, concept, and decision analysis, which contributes to the adaptive cross-task analysis from the discrete sentiment space to the continuous sentiment space. The above joint analysis of the evaluation system and modality number indeed leads to the more flexible and generable multimodal sentiment semantic decoding paradigm. The experiments demonstrate that our sentiment semantic analysis network can achieve state-of-the-art performance.