The Goofy Game: an Approach to Medical AI Misalignment

While Large Language Models (LLMs) offer transformative potential across domains, often outperforming human benchmarks in various tasks, they remain vulnerable to exploitation by users aiming to override their safety protocols. Despite the progress achieved through red teaming methodologies in uncovering and mitigating such vulnerabilities, one notably persistent technique, referred to here as the “Goofy Game” , which leverages role-playing strategies, continues to bypass many existing safeguards. This technique can elicit unsafe responses from LLMs, which, although seemingly benign in isolation, could lead to severe consequences when deployed within high-stakes environments such as clinical decision-making or patient communication. In this study, we build on the insights from our previous exploratory experiments and analyse how a malicious user, even without technical knowledge of the internal architecture and parameters of generative AI models, could create a role-playing prompt that coerces a language model (LLM) into generating incorrect and potentially harmful clinical suggestions. Our objective is to elucidate a particular vulnerability scenario and provide insights that will contribute to future advancements in the development of secure and reliable AI systems.