Towards interpretable echocardiographic diagnosis with prototype-based deep learning

Cardiac diseases including atrial fibrillation (AF) and aortic stenosis (AS) represent major public health challenges, with AF being the most prevalent clinical arrhythmia and AS affecting approximately 5% of individuals over age 65. While deep learning has demonstrated remarkable potential for automated echocardiographic analysis, the black-box nature of these models fundamentally limits their clinical adoption, as healthcare practitioners cannot rely on diagnostic systems whose decision-making processes remain opaque and unverifiable. To address this critical gap, we develop a series of inherently interpretable deep learning methods for echocardiographic diagnosis, specifically focusing on prototype-based approaches that provide transparent, case-based reasoning aligned with clinical decision-making processes. Key challenges in developing interpretable echocardiographic AI include providing transparency in both perception and decision-making, capturing hierarchical multi-scale visual patterns, handling uncertainty and image quality variability, and leveraging complementary information from clinical reports while maintaining inference-time deployability. We introduce methods that address these challenges while preserving the strong predictive performance of deep neural networks. We first demonstrate the limitations of post-hoc explainability through an AF detection model with occlusion-based attribution. We then present ProtoASNet, a prototype-based neural network for AS severity classification that provides inherent interpretability through similarity comparisons with learned spatio-temporal prototypes, while incorporating uncertainty prototypes to flag unreliable inputs. To capture hierarchical feature relationships, we introduce HAPPI, a framework that embeds prototypes in hyperbolic space to organize features from fine-grained local patterns to broader contextual representations. We extend this approach to echocardiography through HiProtoNet, which explicitly models the multi-scale nature of cardiac pathology. Finally, we present MultiProto, a multimodal model that learns from both echocardiography videos and clinical reports during training while requiring only imaging data at inference, enabling richer prototype representations that encode both structural patterns and semantic clinical knowledge. We validate our methods on both private institutional datasets and the public TMED-2 dataset, demonstrating competitive or superior performance relative to existing state-of-the-art approaches while providing clinically meaningful explanations for model predictions.