Abstract This reflection explores the gap between explaining and understanding artificial intelligence’s (AI) outcomes, focusing on the example of AI-driven medical diagnostics. Explainable AI (XAI) is fundamentally rooted in machine learning and probabilistic methods, which is why its relationship to domain-specific contexts, such as medical diagnostics or policymaking, as well as to social concepts, such as trust or accountability, remains ambiguous. This creates a gap between what is technically considered as explaining and what, in turn, constitutes the understanding of a model’s predictions and its concrete, situated implications. In our contribution, we draw on the philosophy of science and its contextual notion of scientific understanding to highlight the need for explainable algorithmic outcomes that enable actors to both explain and intervene in creating, applying or interpreting knowledge. Our reflection highlights the potential tension between XAI as providing insights into an AI model’s technical operations and explainability as part of a situated understanding of how the model works within specific social contexts. We believe that bringing the concept of scientific understanding to empirical AI ethics can help us rethink explainability and contribute to more responsible ways of making sense of and interacting with black-box models, particularly in medical diagnostics.
Gorrieri et al. (Thu,) studied this question.