Multimodal Stress-Aware Ensemble Learning (MSAEL): tri-modal behavioral-physiological fusion for real-time proxy-based stress recognition integrating typing, facial, and voice dynamics

The progression of Real-Time, stress-aware classification using high-arousal affective proxies derived from facial, vocal, and behavioral cues in affective computing is hindered by the fragmentation of human stress expression across emotional, physiological, and cognitive–behavioral domains. Unimodal and bimodal techniques generally fail to capture this multidimensional complexity, leading to limited applicability and unstable predictions in unrestrained settings. In this work, stress is modeled using high-arousal negative emotional states as proxies, and the proposed framework focuses on stress-proxy classification rather than direct physiological stress detection. This study describes Multimodal Stress-Aware Ensemble Learning (MSAEL), a technically sophisticated tri-modal system that incorporates facial emotion patterns, vocal–physiological indicators, and typing-driven behavioral dynamics into a deep ensemble architecture to address this breach. Therefore, to forecast anxiety across Low, Moderate, and High categories, the system uses modality-specific feature extractors, adaptive attention-driven fusion, and a stacked ensemble classifier. In this study, a PRISMA-guided methodological review of 30 multimodal emotional computing research papers informed this framework's methodology. Experimental datasets included FER-2013 (facial expressions), RAVDESS (speech emotion), and CMU Keystroke Dynamics, composed with synthetically aligned tri-modal samples to approximate real-time use. MSAEL surpassed unimodal, bimodal, and advanced multimodal methods, including MDFN, AVEC Fusion, and EAEL, with 91.4% accuracy, 91.3% F1-score, and 0.956 AUC-ROC. Extirpation findings validate each modality's distinct contribution, while correlation and statistical inference studies demonstrate the tri-modal fusion strategy's consistency, reliability, and complementarity. MSAEL is a vigorous stress detection framework that can overcome noise sensitivity, inter-modal discrepancies, and perplexing emotional overlaps. Multimodal affective computing is improved by demonstrating that behavioral, physiological, and expressive signs are most effective for identifying stress. The study demonstrates that MSAEL might be used in occupational well-being nursing, digital medicines, telemedicine, and edge-intelligent human–machine interactions. Unified tri-modal dataset development, physiological wearable integration, edge optimization, and privacy-preserving deployment via federated learning are future goals. Instead of performing subject-level, synchronously recorded, multimodal measurements, the proposed framework seeks to analyze modality-level complementarity using both the model-based methodology and a methodology based on labeled, aligned fusion of independently collected public datasets. Therefore, as on-site synchronized stress sensing of individuals within companies cannot be done, results should not be used to justify its real application. Instead, empirical data on inference latency should be used, as these two studies will not align.