Design, development, and validation of a multimodal synergy-based intuitive virtual and augmented reality therapy platform for mental health

Embodied therapies such as movement therapy have shown promise in enhancing emotional regulation, cognitive engagement, and physical rehabilitation. However, scalable and personalized delivery of such interventions remains a critical challenge. This work presents SIVAM (Synergy-based Intuitive Virtual and Augmented Mental Health platform), a multimodal system that integrates immersive virtual environments, markerless motion capture, physiological sensing, and humanoid robotic mirroring to support affect-aware interventions for mental health. SIVAM combines RGB camera-based skeletal tracking with EEG, EMG, ECG, GSR, and skin temperature sensing using a wearable dry electrode headset to create a closed-loop therapeutic framework. Movement synergies–low-dimensional coordinated patterns across body joints and muscles–are extracted from motion data and aligned with physiological signals to infer affective and motor states in real time, serving as potential biomarkers of stress. The system further introduces a plane-wise movement model that enables natural 3D avatar navigation using a single RGB camera, enhancing embodiment and interaction with virtual environments. A pilot study (N = 5) with five participants of varying dance experience demonstrated reliable motion tracking, real-time synchronization of physiological and movement data, and robust avatar and robot mirroring across diverse movements. These results highlight the feasibility of combining multimodal sensing, virtual avatars, and socially assistive robots to enable scalable, home-based movement therapy.