Estimating residential flexibility from smart meter data: A bilevel framework for demand response aggregation

Existing bilevel optimization frameworks for residential demand response assume known household flexibility characteristics, limiting their applicability to real-world portfolios where flexibility is estimated from consumption data. This paper develops a data-driven bilevel framework that addresses both challenges: estimating flexibility from smart meter data and optimizing aggregator-household coordination under behavioral uncertainty. The estimation component applies an appliance-agnostic decomposition to 15-minute resolution data, extracting flexible loads without requiring device-level metering. These estimates feed into a Stackelberg game where the aggregator optimizes flexpoint-based rewards and households respond by scheduling flexible consumption. The framework captures behavioral heterogeneity by distinguishing automated households providing deterministic responses from manual households exhibiting probabilistic participation with willingness decay. Simulation across 12 German households over 364 days enables 20.7 MWh of shifted load, generating 1029 EUR in cost savings. For this portfolio, participation frequency drove economic performance more strongly than event magnitude, with automated households contributing the majority of total value despite comprising half the portfolio. The framework provides aggregators with a scalable approach to quantify and coordinate distributed residential flexibility. • Appliance-agnostic flexibility estimation from 15-minute smart meter data. • Bilevel Stackelberg framework optimizes aggregator-household coordination. • Behavioral heterogeneity distinguishes automated and manual households. • Portfolio mobilizes 20.7 MWh shifted load generating 1029 EUR savings. • Participation frequency drives performance more than event magnitude.