Heating, ventilation, and air-conditioning (HVAC) systems account for a large share of building energy use, and their operation is strongly shaped by occupant behavior. Occupant-centric control (OCC) can improve HVAC energy efficiency while maintaining thermal comfort, but most existing approaches rely on sparse and limited behavioral signals, such as occupant presence, and miss the occupant’s actual intention and activity context that drive both comfort needs and energy demand. This study introduces an occupant intention-aware framework that integrates multimodal large language models with a model predictive controller. The language model observes camera footage and infers structured occupant and building states, which update the controller's comfort bounds and heat-gain predictions in real time, enabling the MPC controller to optimize heat pump operation more accurately. A 13-hour residential case study in a fully furnished 111 m² residential lab home in Texas demonstrated the effectiveness of the proposed pipeline. A six-configuration case study shows that the proposed framework reduces unmet degree hours by 48.1% and heat-pump electricity use by 60.4% relative to a baseline predictive controller driven by U.S. Department of Energy prototype building schedules in a simulation environment, and progressive ablation indicates that the occupant state is the dominant driver of the improvement. This study underscores the potential of intention-aware OCC, in which LLMs serve as a cognitive layer that translates unstructured observations into structured and explainable semantic states, reducing reliance on fragmented sensor streams and enabling control strategies directly aligned with occupant intentions.
Qi et al. (Mon,) studied this question.