Abstract AR applications are rapidly gaining adoption, expanding users’ perceptual capabilities. Their potential for addressing visual occlusions and effectively extending the user’s line of sight significantly enhances their value in various contexts, from professional to personal use. This work presents a novel system designed to project 3D human poses onto AR glasses, enabling users to perceive concealed individuals behind solid objects, addressing a critical limitation of traditional visual perception. To achieve real-time and accurate 3D projection, we employ fiducial markers strategically placed within the environment. The markers are periodically fused with IMU sensor data to accurately estimate the user’s head orientation, a crucial step for correct spatial alignment. Furthermore, we leverage a multi-view 3D human pose estimation method using calibrated cameras and incorporate attention mechanisms. These mechanisms focus the system on relevant features, improving accuracy and minimizing 3D joint error. Our experiments demonstrate that the proposed framework accurately projects 3D skeletal representations onto AR glasses, even when significant occlusions are caused by solid objects or other occupants within the scene. This novel approach offers a method to enhance situational awareness in dynamic environments where visibility is compromised, potentially benefiting various applications, from first response scenarios to security and surveillance.
Pastaltzidis et al. (Fri,) studied this question.