Why care resists coding: the moral limits of medical AI

Abstract The growing integration of artificial intelligence (AI) into healthcare raises substantial normative questions about the nature of caregiving. While medical AI can improve clinical efficiency and assist with administrative tasks, I argue that patient-facing medical AI cannot straightforwardly substitute for genuine caregiving—understood as an embodied, relational, and morally responsive practice that remains answerable to patients’ vulnerability over time. Drawing on conceptual analysis and empirical evidence, I contend that caregiving exceeds technical competence, requiring context-sensitive responsiveness and a form of moral commitment that grounds responsibility when interpretation and response go wrong. Empirical studies on touch in clinical settings underscore how embodied reassurance and trust are mediated by consent, context, and relational meaning in ways current technological proxies only partially reproduce. Although medical AI may simulate elements of empathy through affective cues and conversational performance, such simulations risk generating interactions that appear supportive while obscuring the locus of responsibility within the care relationship. Hybrid AI-human care models and ‘ethics-by-design’ approaches, while often proposed as solutions, can diffuse accountability and fragment moral responsibility unless governance and clinical workflow design preserve clear human answerability. More critically, increasing reliance on patient-facing medical AI may not only yield inadequate simulations of care, but also contribute to a gradual redefinition of what caregiving entails. As expectations shift, relational responsiveness may be displaced by procedural adequacy as the prevailing standard.