Integrating foundation models with change detection to identify tasking for service robots

Mobile manipulators show incredible promise as domestic service robots – interacting with a wide variety of objects using increasingly affordable hardware. But although perception, manipulation, and mobility have advanced, there remain fundamental challenges in making robots more useful. How can a robot proactively identify tasks that it can complete while supporting individual human preferences for how a home should be configured? We propose using foundation models to first detect what has changed and then select appropriate tasks for the service robot. Change affords action. Only those objects that have been interacted with need to be considered for tasking. Other objects, even if located in non-standard positions in the house, can be ignored. Open-vocabulary based object detection and neural radiance field models are used to identify changes corresponding to fixed phrases. Large language models then validate which tasks should be completed by the robot. Experiments are conducted on data collected by both mobile phone and Stretch 2 Mobile Manipulator, demonstrating general applicability to a wide range of applications in the home.