Context-aware Action Quality Assessment using Latent Regression-based Progressive Sub-action Learning

Most existing Action Quality Assessment (AQA) systems infer performance scores from holistic, video-level representations, a process that frequently obscures the contribution of individual motion segments to the final prediction. This global treatment reduces transparency and limits interpretability, particularly for complex skills composed of multiple sequential phases. The problem is further compounded by limited fine-grained supervision and the difficulty of modeling long-range temporal dependencies across an action sequence. This paper introduces a Latent Regression-based Progressive Sub-action Learning framework to address these limitations by capturing rich contextual information within the action sequence. The framework first employs a procedure-parsing module to segment each video into semantically coherent sub-actions and extract corresponding spatio-temporal features. A latent variable approach subsequently generates initial pseudo-subscores by integrating these features with overall score labels from the training set. Critically, these pseudo-subscores are progressively refined through an iterative process that leverages the cumulative contextual information from preceding sub-actions. This yields an enriched feature set that accurately encapsulates the execution history. Furthermore, a novel monotonic penalty loss is introduced to enforce a logical and consistent progression in the sub-scores, mitigating abrupt and illogical score fluctuations. The model is trained in a two-stage process. First, it generates the robust pseudo-subscores, and subsequently it predicts both the overall score and the per-substage scores. This explicit modeling of long-range dependencies, along with consistent score progression, is critical for ensuring prediction accuracy. Extensive experimental evaluation confirms that this structured modeling approach delivers consistent performance gains over contemporary AQA methods while offering substantially improved interpretability.