Explainable Sequential Deep Learning for Detecting Botnet Command-and-Control Communications in Transport Layer Security-Encrypted Network Traffic

Darlington Nwanebu, E. H. Gadzama, Shamsu Garba

Abstract


The increasing use of Transport Layer Security (TLS) encryption has improved communication privacy but has also created challenges for detecting botnet Command-and-Control (C2) communications. This study presents an explainable sequential deep-learning framework combining Bidirectional Long Short-Term Memory (BiLSTM) and Transformer models for detecting botnet C2 communications in TLS-encrypted network traffic. Random Forest was used as a baseline. The study considered CTU-13, CIC-Encrypted-Traffic-2022, and a synthetically generated TLS-BotFlow2025 dataset. Five-fold cross-validation produced F1-scores of 0.74, 0.84, and 0.87 for Random Forest, BiLSTM, and Transformer, respectively, with ROC-AUC values of 0.81, 0.88, and 0.92. The hybrid model achieved a best validation F1-score of 0.9640 and ROC-AUC of 0.9870 at epoch 47. These are validation results, not independent test-set results. External evaluation on unseen CTU-13 traffic produced F1-scores of 0.67, 0.75, and 0.80 for Random Forest, BiLSTM, and Transformer, respectively. SHAP and attention analysis were used to support interpretation of model decisions. 


Full Text:

PDF

References


REFERENCES

Ahmed, U., Jiangbin, Z., Almogren, A., Sadiq, M., Rehman, A. U., Sadiq, M. T., & Choi, J. (2024). Hybrid bagging and boosting with SHAP based feature selection for enhanced predictive modeling in intrusion detection systems. Scientific Reports, 14, Article 30532. https://doi.org/10.1038/s41598-024-81151-1

Ferrag, M. A., Ndhlovu, M., Tihanyi, N., Cordeiro, L. C., Debbah, M., Lestable, T., & Thandi, N. S. (2024). Revolutionizing cyber threat detection with large language models: A privacy-preserving BERT-based lightweight model for IoT/IIoT devices. IEEE Access, 12, 23733–23750. https://doi.org/10.1109/ACCESS.2024.3363469

Fu, C., Li, Q., & Xu, K. (2023). Detecting unknown encrypted malicious traffic in real time via flow interaction graph analysis. In 30th Annual Network and Distributed System Security Symposium (NDSS 2023). Internet Society. https://doi.org/10.14722/ndss.2023.23080

Hashmi, A., Barukab, O. M., & Hamza Osman, A. (2024). A hybrid feature weighted attention based deep learning approach for an intrusion detection system using the random forest algorithm. PLOS ONE, 19(5), e0302294. https://doi.org/10.1371/journal.pone.0302294

Hossain, M. A., & Islam, M. S. (2023). A novel hybrid feature selection and ensemble-based machine learning approach for botnet detection. Scientific Reports, 13, Article 21207. https://doi.org/10.1038/s41598-023-48230-1

Kumar, M., Kondaiah, C., Pais, A. R., & Rao, R. S. (2023). Machine learning models for phishing detection from TLS traffic. Cluster Computing, 26(5), 3263–3277. https://doi.org/10.1007/s10586-023-04042-6

Wang, K., Gao, J., & Lei, X. (2023). MTC: A multi-task model for encrypted network traffic classification based on Transformer and 1D-CNN. Intelligent Automation & Soft Computing, 37(1), 619–638. https://doi.org/10.32604/iasc.2023.036701

Yang, J., Jiang, X., Lei, Y., Liang, W., Ma, Z., & Li, S. (2024). MTSecurity: Privacy-preserving malicious traffic classification using graph neural network and Transformer. IEEE Transactions on Network and Service Management, 21(3), 3583–3597. https://doi.org/10.1109/TNSM.2024.3383851

Yin, Z., Qin, R., Ye, C., Li, Y., He, Y., Shu, Y., & Jiang, R. (2022). Dilated convolution-based botnet detection model. In C. Zhao & H. Imane (Eds.), Third International Conference on Computer Communication and Network Security (CCNS 2022) (Vol. 12453, Article 124531D). SPIE. https://doi.org/10.1117/12.2659107.


Refbacks

  • There are currently no refbacks.