With the rapid development of digital-intelligence service platforms, web-based applications are increasingly required to provide real-time and reliable AI-driven services under diverse and dynamic workloads. However, the coexistence of multiple heterogeneous models and the variability of user requests pose critical challenges for efficient model scheduling and inference optimization. To address these issues, we propose MAIS (Multi-model Adaptive Inference Scheduling), a unified framework that integrates workload-aware multi-model scheduling with adaptive inference strategies. Specifically, MAIS dynamically allocates computational resources and selects appropriate inference modes by jointly considering user latency requirements, platform resource availability, and model complexity. Its core innovation lies in combining these factors in a unified optimization framework while also supporting hierarchical inference optimization, which allows dynamic adaptation to varying workloads and model heterogeneity. Furthermore, it supports hierarchical inference optimization through model compression, selective execution, and edge-cloud collaboration, achieving a balance between accuracy, efficiency, and service quality. Extensive experiments demonstrate that MAIS significantly improves response latency, resource utilization, and service robustness compared to existing scheduling and inference methods, making it well-suited for next-generation digital-intelligence service platforms.
Li et al. (Wed,) studied this question.