Adaptive Dexterous Manipulation for Humanoid Robots via a Guided Tactile Multimodal Foundation Model

Key Points

• The research aims to address challenges in precise grasping for humanoid robots by utilizing a guided tactile multimodal framework.
• Developed the Guided Tactile Multimodal Framework (GTMF) to address perception blind spots and control instabilities.
• Utilized vision and tactile sensory data for real-time grasp adjustment in humanoid robots.
• Implemented a wearable data collection module to integrate tactile and visual data for analysis.
• Applied a guided mask attention network for pose calibration and cross-modal feature alignment.
• The framework achieved improved grasping accuracy compared to existing methods using only visual inputs.
• Demonstrated the ability to generate tactile feedback from RGB images for real-time adjustments.
• Validated the effectiveness of the method through various experimental tests.

Abstract