SemanticAction: a Semantic-Driven and Behavior-Aware Password Framework for Adaptive VR Authentication under Observation Attacks

As immersive Virtual Reality (VR) applications become increasingly widespread, ensuring secure and usable authentication is critical. Traditional knowledge-based methods (e.g., passwords, PINs) suffer from memorability issues and are highly vulnerable to observation attacks such as Man-in-the-Room (MITR). Meanwhile, biometric and behavioral approaches raise concerns regarding practicality, privacy, and cross-platform deployment. We present SemanticAction, a semantic-driven, behavior-augmented authentication framework that integrates knowledge-based passwords with gesture-based behavioral biometrics. Passwords are encoded as scene anchored directions executed through intuitive hand gestures, enabling semantic meaning to guide user interactions. To counter observation attacks, SemanticAction employs randomized scene prompts and decoy scenes, while a dual-constraint verification mechanism adapts to both cold-start/few-shot conditions. Two user studies provide initial evidence that SemanticAction can mitigate MITR attacks while alleviating memorability challenges, maintaining favorable usability and security performance even with limited behavioral data. This work offers early insights and practical design considerations for behavior-aware authentication in VR.