Traffic analysis for secure communication: robust DoH tunnel detection via temporal convolutional network with semantic modulation and ReLU attention

Traffic analysis is a cornerstone of secure communication in modern network systems, and the accurate transmission of traffic semantics plays a critical role in this process. DNS over HTTPS (DoH) protocol was designed to enhance user communication privacy by encrypting DNS traffic. However, attackers can abuse DoH to encrypt malicious DNS tunnel traffic, making traditional detection methods ineffective. Previous DoH tunnel detection approaches suffer from limited feature utilization and struggle to identify malicious flows, especially in class-imbalanced scenarios. In this paper, we propose a model named STCR-DoH, which effectively addresses these issues. Specifically, we introduce a semantic modulation layer to amplify subtle malicious traffic semantics and suppress irrelevant noise. We then employ a temporal convolutional network to efficiently capture semantic information in parallel and introduce a ReLU attention mechanism to learn long-range dependencies positively correlated with malicious semantics. With only 19 important features as input, our model achieves 99.91% accuracy and 99.68% recall in class-imbalanced detection scenario, surpassing eight state-of-the-art methods.