What question did this study set out to answer?

The aim is to address the adverse effects of multimodal interactions on prediction performance by developing a novel deep learning strategy.

March 19, 2026

View Full Paper

When Multimodal Interactions Impair Prediction: A Novel Regularized Deep Learning Strategy

GCGang ChenFudan University SXShuaiyong XiaoTongji University CZCheng ZhangChina Meteorological Administration

Key Points

The aim is to address the adverse effects of multimodal interactions on prediction performance by developing a novel deep learning strategy.
Conceptualize multimodal interactions to understand their effects on predictions
Propose a strategy named NIRMD to regularize negative interactions in deep learning
Conduct empirical evaluation in case studies for various prediction tasks
NIRMD improves the prediction performance of existing multimodal deep learning methods
Ablation and feature contribution analyses provide evidence of NIRMD's effectiveness
Positive and negative interactions are effectively managed using the proposed approach

Abstract

Multimodal data are proliferating and hence flourishing data-driven business decision making, exemplified by short video attractiveness prediction (SVAP), multimodal review sentiment classification (MRSC), and multimodal data-based default risk prediction (DRP). However, when data of various modalities (e.g., text, graph, image, and video) are used jointly, they may mutually interact, adversely affecting prediction performance. To unravel and resolve the opaque conflicts in multimodal data, we formally conceptualize multimodal interactions and provide analytical insights for mitigating negative interactions at the feature, modality, and modality-wise instance levels. To better realize the predictive power of multimodal data, we propose a novel deep learning strategy named NIRMD (for negative interaction-regularized multimodal deep learning), which allows positive (negative) multimodal interactions to be effectively encouraged (mitigated) in a learnable nonlinear representation space. Empirical evaluation in three case studies involving SVAP, MRSC, and DRP, respectively, shows that the prediction performance of state-of-the-art multimodal deep learning methods can be enhanced by incorporating NIRMD. Exploratory (i.e., ablation, feature contribution, and case) analyses render evidence of NIRMD’s effectiveness in mitigating negative multimodal interactions. History: Accepted by Ram Ramesh, Area Editor for Data Science & Machine Learning. Funding: G. Chen was supported by the National Natural Science Foundation of China Grants 72522010, 72301239, and 72394371. S. Xiao was supported by the National Natural Science Foundation of China Grants 72301194, 72495133, and 72472058. C. Zhang was supported by the National Natural Science Foundation of China Grants 72271059 and 72571071. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2024.0794 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2024.0794 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

AI에게 질문

Bookmark

View Full Paper

AI에게 질문

Bookmark

View Full Paper

When Multimodal Interactions Impair Prediction: A Novel Regularized Deep Learning Strategy

Key Points

Abstract

Cite This Study