Dual-Channel Mamba-Based Semantic–Behavioral Feature Learning with Prototype-Guided Zero-Shot Inference for Zero-Day Malware Detection

Detecting previously unseen malware remains a critical challenge for modern cybersecurity systems due to the rapid evolution of malicious software and the limitations of traditional supervised detection models. This paper proposes a Dual-Channel Mamba-Based Semantic–Behavioral Feature Learning framework for zero-day malware detection that jointly models static malware artifacts and dynamic execution traces within a unified representation space. The proposed architecture employs two parallel encoders that extract semantic features from executable structures and behavioral features from API call sequences. These features are integrated through a cross-channel fusion mechanism and processed using a Mamba-based selective state space architecture, which efficiently captures long-range dependencies in malware behavior while maintaining linear computational complexity. To address the zero-day detection problem, a prototype-guided inference strategy is introduced that enables similarity-based classification of previously unseen malware families within the learned embedding space. Extensive experiments conducted on multiple malware datasets demonstrate that the proposed framework consistently outperforms strong deep learning baselines. The model achieves an average classification accuracy of 96.01% ± 0.35 and an F1-score of 95.56% ± 0.36, while the zero-day detection rate reaches 88.93% ± 0.98, significantly improving detection performance compared with transformer and recurrent architectures. Runtime analysis further shows that the proposed framework achieves efficient inference with an average latency of approximately 8 ms per sample, making it suitable for real-time malware analysis systems. These results indicate that combining dual-channel feature learning with Mamba-based sequential modeling provides an effective and scalable solution for detecting both known and previously unseen malware threats.