High-fidelity marker-level 3D deformation simulation of visuotactile sensors

The visuotactile simulation is significant for robot learning towards contact-rich manipulation tasks. However, tactile simulation is still a challenging problem in most robot simulators. In this paper, we introduce a marker-level 3D deformation simulation of visuotactile sensors utilizing the SOFA framework, which employs a Finite Element Method (FEM) to simulate the contact dynamics of the sensor’s soft elastomer. To achieve high-fidelity simulation, a sim-real differences-based optimization method is proposed to obtain the physical simulation parameters using aligned simulated and real tactile markers. We also introduce contact-aware dynamic material parameters to enhance the simulation consistency under various contact states. Furthermore, we develop an in-hand object pose estimation dataset in this simulation, containing 24000 tactile 3D point clouds and ground truth samples, and release it to the community for in-depth studies in object pose estimation. In addition, we conduct extensive experiments to verify the consistency of simulated and real tactile signals. The results show that the proposed visuotactile simulation could produce high-fidelity tactile markers in 3D space and capture the tactile response under various dynamic contact.