Sensor fusion of touch & vision in soft manipulators for fruit picking

Modern agricultural robotic systems are constrained by limited sensing and manipulation capabilities, particularly in fruit harvesting, where variability in color, size, and firmness poses significant challenges. Existing solutions, often reliant on rigid grippers and single-modality sensors, frequently cause fruit bruising and substantial postharvest losses. Here, we present a compact, five-finger soft robotic gripper with integrated multimodal sensing—including vision, tactile, and curvature sensing—for adaptive and non-destructive fruit harvesting. The system incorporates 13 sensors, onboard electronics, local computation, and a rotational harvesting module. Each finger embeds custom stretchable optical fibers that function as tactile and curvature sensors, while the palm houses a miniaturized camera and distance sensor. The gripper actuates within two seconds at 80 kPa, exerts up to 6 N of pulling force, and lifts objects up to 1 kg—more than 16 times its own weight. Its workspace expands from 200 mm² to 14,000 mm², enabling the handling of fruits with diverse shapes and sizes. Each finger bends up to 240°, with performance closely matching finite element predictions. For vision measurements, the hue channel in the HSV color space enables robust real-time color detection, achieving 100% shape classification accuracy and a size measurement error below 1.8%. Tactile sensors distinguish soft from firm objects, while curvature sensors accurately measure the finger’s bending state—both based on optical signal loss. Real-time demonstrations validate the system’s ability to assess ripeness using multimodal data (vision, tactile, and curvature) and successfully harvest greenhouse strawberries with minimal damage. This platform offers a versatile, sensor-rich solution for both precision agriculture and general-purpose robotic manipulation. The authors report multifunctional and multimodal sensing embodied in a soft gripper with integrated vision and touch that measures fruit size, shape and firmness, accurately assessing ripeness and harvesting strawberries and delicate fruits.