Mamba-ACP: a Hybrid State-Space and Transformer Framework for Interpretable Anticancer Peptide Prediction

Anticancer peptides (ACPs) represent a promising class of therapeutic agents that selectively destroy cancer cells while sparing healthy tissues. Despite their potential, biological challenges-including poor biochemical stability, limited tumor selectivity, and inefficient delivery mechanisms - hinder their clinical translation. In parallel, the rapid expansion of peptide sequence data underscores the urgent need for accurate, scalable, and generalizable ACP prediction models. To address these limitations, we propose a robust hybrid deep learning framework-Mamba-ACP-that integrates transformer-based Evolutionary Scale Modeling (ESM-2) embeddings, handcrafted features (AAindex, BLOSUM62), and a Mamba-based sequence modeling architecture. This approach captures both evolutionary and physicochemical properties of peptides to enhance prediction performance. The model was trained and validated by using two benchmark datasets Set 1 and Set 2, commonly used in peptide-based computational biology. Mamba-ACP achieves 87.59% accuracy and an AUC of 0.9356 on Set 1, and 96.69% accuracy and an AUC of 0.9922 on Set 2, beating state-of-the-art ACP predictors like ACP-CapsPred and GRDF by a significant margin. The Mamba-ACP framework processes a token-level fused representation obtained by concatenating the token-level ESM-2 embeddings with the PCA-reduced handcrafted residue descriptors at each sequence position. These results affirm the effectiveness of combining pre-trained transformer embeddings with handcrafted features and structured sequence modeling in improving ACP classification. Our findings position Mamba-ACP as a new benchmark in computational peptide discovery, offering strong generalizability, reduced false positives, and efficient performance. We further provide model-level explanations via gradient-based residue/token saliency, SHAP feature importance for AAindex/BLOSUM62 descriptors, motif-level enrichment, and saliency-guided residue mutation validation.