When Human-in-the-Loop Fails: An Answerability Test for Deployed AI Systems

This paper develops a structural account of why human-in-the-loop oversight often fails in deployed AI systems. Its central claim is that the presence of a human somewhere in the process does not by itself secure answerability. In many real deployments, the human role is procedural rather than consequence-bearing, while the system materially steers outcomes through ranking, filtering, scoring, routing, summarization, drafting, or recommendation. Under these conditions, the visible language of oversight remains intact, but the underlying answerability structure weakens. The paper argues that deployed AI systems become unsafe when they acquire steering power without inheriting proportionate consequence, and when the speed, scale, opacity, or workflow structure of execution exceeds the human capacity for meaningful witness and timely correction. It introduces an Answerability Test for Deployed AI Systems, according to which a deployment fails when it combines material steering without proportionate consequence, procedural oversight without real witness, liability laundering, override-friction asymmetry, severance by speed or opacity, exportable error cost, and the appearance of control without corrigibility. The paper’s broader contribution is to show that the deepest failure in many contemporary deployments is not simply technical misbehavior, but the collapse of answerability behind the visible layer of human oversight. Human-in-the-loop does not secure safety where the human has already ceased to function as a real witness.