DQN ‐Guided Subset‐Induced OCSVM Kernel Approximation for Imbalanced Anomaly Detection

Anomaly detection under limited normal data remains a fundamental challenge due to severe class imbalance and scarcity of anomalies. We propose a novel framework that reformulates support vector selection in One‐Class SVM as a sequential decision‐making problem. Based on the intuition of OCSVM, we treat support vectors as an a priori assumption for kernel approximation, rather than a posterior outcome of optimization. A Deep Q‐Network (DQN) agent adaptively selects representative subsets guided by validation feedback, thereby inducing a task‐aware low‐rank kernel approximation, while ridge‐regularized weight recovery ensures numerical stability. Extensive experiments on multiple tabular benchmarks covering diverse imbalanced anomaly detection scenarios show that the proposed method consistently outperforms traditional methods, deep learning models, and LLM‐based baselines in both ranking and classification metrics, demonstrating particularly strong generalization to unseen data and across diverse datasets. © 2026 The Author(s). IEEJ Transactions on Electrical and Electronic Engineering published by Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.