Application of representation learning in detecting botnet attacks

Botnet detection remains a perennial and critical challenge in cybersecurity. As long as the internet exists, threat actors will devise new ways to create and disguise these malicious networks, making the development of robust detection methods a task that will never be obsolete. Traditional approaches, relying on rigid signatures and manual feature engineering, are often locked in a reactive cycle. A more critical limitation is their poor generalization; models trained on known botnets frequently fail to detect novel, unseen threats, rendering them vulnerable in real-world scenarios. This paper introduces a robust framework that significantly enhances botnet detection by overcoming these limitations. We propose a novel methodology that combines advanced feature engineering, such as octet splitting for IP addresses, with a sophisticated representation learning technique using the Hilbert space-filling curve to transform network flows into 2D images. This approach preserves data locality and eliminates the noise introduced by traditional zero-padding. Furthermore, we address the critical issue of class imbalance using a combination of SMOTE, a weighted sampler, and Focal Loss to focus the model on more challenging samples. To rigorously evaluate the model's real-world applicability, we employed a challenging cross-scenario validation strategy, training the model on the Murlo botnet (Scenario 8) and testing it on the completely unseen Rbot botnet (Scenario 10) from the publicly available CTU-13 dataset. Our proposed model achieved an outstanding accuracy of 98.34% and a weighted F1-score of 98.38%, demonstrating a remarkable ability to generalize to novel botnet attacks. This result validates our approach and highlights the superiority of learned, spatially-aware representations over traditional models, which failed to detect the unseen botnet. Our work presents a significant step towards creating more adaptive and resilient intrusion detection systems capable of handling novel, unseen botnet families.