MERAC: Multimodal fusion of ECG and clinical report with local–global encoding for arrhythmia classification

Electrocardiogram (ECG) analysis plays a crucial role in arrhythmia detection. However, traditional self-supervised learning methods that focus exclusively on ECG data often struggle to capture rich semantic information, resulting in suboptimal feature representations. This paper aims to address this limitation by improving the performance of arrhythmia classification through multimodal learning. We introduce MERAC (Multimodal Fusion of ECG and Clinical Report with Local-Global Encoding for Arrhythmia Classification), a novel framework that leverages both local and global encoding modules to enhance semantic transfer across modalities. The local encoding module employs fine-grained local alignment to address the coarse semantic granularity inherent in traditional global alignment approaches. Simultaneously, the global encoding module incorporates a text-guided probabilistic masking strategy to promote comprehensive feature fusion and achieve richer semantic representations. MERAC demonstrates superior performance compared to existing state-of-the-art methods. On the CPSC2018 dataset, our approach achieves a zero-shot AUC of 89.73% and an F1 score of 56.39%. On the Chapman dataset, it attains an AUC of 90.06% and an F1 score of 61.50%. The experimental results highlight the effectiveness of our method in cardiac arrhythmia detection. The proposed MERAC framework successfully integrates ECG signals with medical reports through innovative local and global encoding strategies, demonstrating significant improvements in classification performance.