One of the key challenges in Multimodal Sentiment Analysis (MSA) is data missing, with existing research primarily focusing on optimizing model architectures to handle missing data scenarios. However, these methods often overlook the potential shared information between noisy auxiliary modalities (audio and visual) and fail to effectively model the complex dependencies among hierarchical representations. To address these challenges, we propose HiGraMI, a hierarchical graph fusion network incorporating mutual information (MI) enhancement. The model tackles the issues mentioned above through two complementary components. First, the Cross-modal MI Enhancement (CMIE) module generates stable auxiliary representations by maximizing MI between audio and visual features. These auxiliary representations are then used to enhance the quality of text representations adaptively through a dynamic adjustment strategy, improving their robustness. Furthermore, the Hierarchical Graph Attention Fusion (HGAF) module treats hierarchical and cross-modal representations as nodes in a graph. It then uses a Graph Attention Network (GAT) to capture interactions between cross-level and cross-modal representations under data missing scenarios. This process generates robust multimodal joint representations, which significantly improve the model’s robustness and sentiment prediction performance. Experimental results demonstrate that HiGraMI outperforms current mainstream baseline models across most metrics and data missing scenarios on the MOSI and MOSEI datasets, achieving state-of-the-art performance.
Zheng et al. (Thu,) studied this question.