A CLIP-based multimodal model predicted LVEF <50 with AUROC 0.83 and E/E' >15 with AUROC 0.77 by embedding ECG and chest X-ray data in a shared space.
Does a CLIP-based multimodal model using ECG and chest X-rays accurately predict LVEF < 50 and E/E' > 15 in cardiology patients?
A multimodal AI model using the CLIP framework successfully predicted echocardiographic parameters of heart failure (LVEF < 50 and E/E' > 15) from concurrent ECG and chest X-ray data.
Absolute Event Rate: 0% vs 0%
Abstract Background In clinical practice, physicians integrate multiple diagnostic tests to infer pathophysiology, but no AI models have been developed to replicate this complex decision-making process. A model capable of interpreting multimodal data within a unified space is required. For instance, in heart failure, chest X-rays and ECGs, while representing different modalities, both reflect the same underlying pathological condition. Purpose This study aims to develop a multimodal model leveraging the CLIP framework to classify heart failure pathophysiology by embedding data from distinct modalities into a shared representational space. Methods We analyzed data from cardiology patients at our University between January 2012 and December 2022, selecting only those who had concurrent ECG, chest X-rays, and echocardiography. The dataset comprised 9,632 patients, with 34,747 12-lead ECGs and 36,366 chest X-rays. CLIP was employed to pair ECGs and chest X-rays from the same patients. Using these vectors, the model was trained to predict LVEF and E/E' values from transthoracic echocardiograms. The model then predicted the classification of each criteria, based on the cut-off values of 50 for LVEF and 15 for E/E'. Results Dimensionality reduction of the vectors obtained from ECGs and chest X-rays demonstrated that both modalities were projected within the same representational space. Figure 1 shows the result of embedding ECG and chest X-ray data into a common feature space using CLIP, with dimensionality reduced to two dimensions using UMAP. Each point in the figure represents a single ECG or chest X-ray data sample, color-coded based on BNP values transformed into a log scale. Samples with high BNP values were distributed in the upper right, while those with low BNP values were located toward the lower left. This indicates that the pathophysiology of heart failure, as inferred from ECGs and chest X-rays, was successfully reproduced within this feature space. In the Figure 1, ECG and chest X-ray data were plotted with overlapping distributions, demonstrating that these modalities can be represented within the same feature space. The feature vectors obtained through CLIP were treated equivalently for ECG and chest X-rays, and the model was trained to predict E/E' and LVEF values. As a result, as shown in Figure 2, the model achieved an AUROC of 0.83 for predicting LVEF 50 and an AUROC of 0.77 for predicting E/E' 15. These results suggest that our model successfully classified the pathophysiology of heart failure. Conclusions This study demonstrates that the CLIP model can successfully classify heart failure pathophysiology by embedding multimodal data into a unified representational space.Figure1 Figure2
Shibata et al. (Sat,) reported a other. A CLIP-based multimodal model predicted LVEF <50 with AUROC 0.83 and E/E' >15 with AUROC 0.77 by embedding ECG and chest X-ray data in a shared space.