A Comprehensive Systematic Review of AI-based Network Intrusion Detection Systems: Techniques, Datasets, Challenges, and Future Research Directions
DOI:
https://doi.org/10.71426/jcdt.v2.i1.pp140-150Keywords:
Network security, Intrusion detection, Artificial intelligence, Deep learning, Cybersecurity, Anomaly detection, Federated learningAbstract
The rapid expansion of cloud computing platforms, Internet of Things (IoT) ecosystems, edge devices, software-defined networks, and distributed enterprise infrastructures has significantly increased the complexity and scale of modern cybersecurity environments. Traditional signature-based intrusion detection systems are increasingly ineffective against sophisticated cyber threats such as zero-day attacks, advanced persistent threats, ransomware propagation, botnet activities, and encrypted malicious traffic because they rely heavily on predefined attack signatures and static rule-based detection mechanisms. Consequently, AI driven intrusion detection systems have emerged as promising solutions for intelligent threat detection, adaptive cybersecurity analytics, and real-time network defense. This paper presents a comprehensive systematic literature review of AI based network intrusion detection systems with particular emphasis on machine learning, deep learning, federated learning, and intelligent anomaly detection frameworks. Furthermore, the study evaluates benchmark cybersecurity datasets including NSL-KDD, CICIDS2017, UNSW-NB15, DARPA, and TON-IoT datasets to investigate their effectiveness, scalability, realism, and applicability for modern intrusion detection research. Comparative analysis indicates that deep learning and hybrid AI techniques significantly improve intrusion detection accuracy, adaptive threat detection, and real-time cybersecurity analytics compared with traditional approaches. The paper further discusses major challenges including adversarial attacks, encrypted traffic analysis, dataset imbalance, explainability, computational overhead, and concept drift. Emerging research directions such as federated intrusion detection, Transformer-based cybersecurity, graph neural networks, self-supervised learning, and explainable AI are also critically analyzed.
References
[1] Denning DE. An intrusion-detection model. IEEE Transactions on Software Engineering. 1987;SE-13(2):222-232. Available from: https://doi.org/10.1109/TSE.1987.232894
[2] Lunt TF. A survey of intrusion detection techniques. Computers & Security. 1993;12(4):405-418. Available from: https://doi.org/10.1016/0167-4048(93)90029-5
[3] Modi C, Patel D, Borisaniya B, Patel H, Patel A, Rajarajan M. A survey of intrusion detection techniques in cloud. Journal of Network and Computer Applications. 2013;36(1):42-57. Available from: https://doi.org/10.1016/j.jnca.2012.05.003
[4] Roesch M. Snort: Lightweight intrusion detection for networks. Proceedings of the 13th USENIX Conference on System Administration (LISA ’99). 1999:229-238.
[5] Garcia-Teodoro P, Diaz-Verdejo J, Maciá-Fernández G, Vázquez E. Anomaly-based network intrusion detection: Techniques, systems and challenges. Computers & Security. 2009;28(1-2):18-28. Available from: https://doi.org/10.1016/j.cose.2008.08.003
[6] Sommer R, Paxson V. Outside the closed world: On using machine learning for network intrusion detection. Proceedings of the 2010 IEEE Symposium on Security and Privacy. 2010:305-316. Available from: https://doi.org/10.1109/SP.2010.25
[7] Buczak AL, Guven E. A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys & Tutorials. 2016;18(2):1153-1176. Available from: https://doi.org/10.1109/COMST.2015.2494502
[8] Kim G, Lee S, Kim S. A novel hybrid intrusion detection method integrating anomaly detection with misuse detection. Expert Systems with Applications. 2014;41(4):1690-1700. Available from: https://doi.org/10.1016/j.eswa.2013.08.066
[9] Yin C, Zhu Y, Fei J, He X. A deep learning approach for intrusion detection using recurrent neural networks. IEEE Access. 2017;5:21954-21961. Available from: https://doi.org/10.1109/ACCESS.2017.2762418
[10] Shone N, Ngoc TN, Phai VD, Shi Q. A deep learning approach to network intrusion detection. IEEE Transactions on Emerging Topics in Computational Intelligence. 2018;2(1):41-50. Available from: https://doi.org/10.1109/TETCI.2017.2772792
[11] Vinayakumar R, Soman KP, Poornachandran P. Applying deep learning approaches for network traffic prediction. Proceedings of the 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI). 2017:2353-2358. Available from: https://doi.org/10.1109/ICACCI.2017.8126198
[12] Apruzzese G, Colajanni M, Ferretti L, Guido A, Marchetti M. On the effectiveness of machine and deep learning for cyber security. Proceedings of the 2018 10th International Conference on Cyber Conflict (CyCon). 2018:371-390. Available from: https://doi.org/10.23919/CYCON.2018.8405026
[13] Mukkamala S, Janoski G, Sung A. Intrusion detection using neural networks and support vector machines. Proceedings of the IEEE International Joint Conference on Neural Networks. 2002;2:1702-1707. Available from: https://doi.org/10.1109/IJCNN.2002.1007774
[14] Lee W, Stolfo S. Data mining approaches for intrusion detection. Proceedings of the 7th USENIX Security Symposium. 1998. Available from: https://dl.acm.org/doi/10.5555/1267549.1267555
[15] Breiman L. Random forests. Machine Learning. 2001;45(1):5-32. Available from: https://link.springer.com/article/10.1023/A:1010933404324
[16] Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation. 1997;9(8):1735-1780. Available from: https://doi.org/10.1162/neco.1997.9.8.1735
[17] Hundman K, Constantinou V, Laporte C, Colwell I, Soderstrom T. Detecting spacecraft anomalies using LSTMs and nonparametric dynamic thresholding. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018:387-395. Available from: https://doi.org/10.1145/3219819.3219845
[18] Sakurada M, Yairi T. Anomaly detection using autoencoders with nonlinear dimensionality reduction. Proceedings of the MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data Analysis. 2014:4-11. Available from: https://dl.acm.org/doi/abs/10.1145/2689746.2689747
[19] Yang Q, Liu Y, Chen T, Tong Y. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology. 2019;10(2):1-19. Available from: https://doi.org/10.1145/3298981
[20] McMahan B, Moore E, Ramage D, Hampson S, y Arcas BA. Communication-efficient learning of deep networks from decentralized data. Proceedings of Artificial Intelligence and Statistics. 2017:1273-1282. Available from: https://doi.org/10.48550/arXiv.1602.05629
[21] Goodfellow IJ, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. arXiv Preprint arXiv:1412.6572. 2014. Available from: https://doi.org/10.48550/arXiv.1412.6572
[22] Papernot N, McDaniel P, Sinha A, Wellman MP. SoK: Security and privacy in machine learning. Proceedings of the 2018 IEEE European Symposium on Security and Privacy (EuroS&P). 2018:399-414. Available from: https://doi.org/10.1109/EuroSP.2018.00035
[23] Mirsky Y, Doitshman T, Elovici Y, Shabtai A. Kitsune: An ensemble of autoencoders for online network intrusion detection. arXiv Preprint arXiv:1802.09089. 2018. Available from: https://doi.org/10.48550/arXiv.1802.09089
[24] Tavallaee M, Bagheri E, Lu W, Ghorbani AA. A detailed analysis of the KDD CUP 99 data set. Proceedings of the 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications. 2009:1-6. Available from: https://doi.org/10.1109/CISDA.2009.5356528
[25] Sharafaldin I, Lashkari AH, Ghorbani AA. Toward generating a new intrusion detection dataset and intrusion traffic characterization. Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP). 2018;1:108-116. Available from: https://doi.org/10.5220/0006639801080116
[26] Moustafa N, Slay J. UNSW-NB15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). Proceedings of the 2015 Military Communications and Information Systems Conference (MilCIS). 2015:1-6. Available from: https://doi.org/10.1109/MilCIS.2015.7348942
[27] Patcha A, Park JM. An overview of anomaly detection techniques: Existing solutions and latest technological trends. Computer Networks. 2007;51(12):3448-3470. Available from: https://doi.org/10.1016/j.comnet.2007.02.001
[28] Lippmann RP, Fried DJ, Graf I, Haines JW, Kendall KR, McClung D, Weber D, Webster SE, Wyschogrod D, Cunningham RK, Zissman MA. Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. Proceedings of DARPA Information Survivability Conference and Exposition (DISCEX). 2000;2:12-26. Available from: https://doi.org/10.1109/DISCEX.2000.821506
[29] Wang W, Zhu M, Zeng X, Ye X, Sheng Y. Malware traffic classification using convolutional neural network for representation learning. Proceedings of the 2017 International Conference on Information Networking (ICOIN). 2017:712-717. Available from: https://doi.org/10.1109/ICOIN.2017.7899588
[30] Javaid A, Niyaz Q, Sun W, Alam M. A deep learning approach for network intrusion detection system. EAI Endorsed Transactions on Security and Safety. 2016;3(9):21. Available from: https://doi.org/10.4108/eai.3-12-2015.2262516
[31] Alrawashdeh K, Purdy C. Toward an online anomaly intrusion detection system based on deep learning. Proceedings of the 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA). 2016:195-200. Available from: https://doi.org/10.1109/ICMLA.2016.0040
[32] Diro AA, Chilamkurti N. Distributed attack detection scheme using deep learning approach for Internet of Things. Future Generation Computer Systems. 2018;82:761-768. Available from: https://doi.org/10.1016/j.future.2017.08.043
[33] Vinayakumar R, Alazab M, Soman KP, Poornachandran P, Al-Nemrat A, Venkatraman S. Deep learning approach for intelligent intrusion detection system. IEEE Access. 2019;7:41525-41550. Available from: https://doi.org/10.1109/ACCESS.2019.2895334
[34] Ring M, Wunderlich S, Scheuring D, Landes D, Hotho A. A survey of network-based intrusion detection data sets. Computers & Security. 2019;86:147-167. Available from: https://doi.org/10.1016/j.cose.2019.06.005
[35] Khan MA, Karim MR, Kim Y. A scalable and hybrid intrusion detection system based on the convolutional-LSTM network. Symmetry. 2019;11(4):583. Available from: https://doi.org/10.3390/sym11040583
[36] Ferrag MA, Maglaras L, Moschoyiannis S, Janicke H. Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study. Journal of Information Security and Applications. 2020;50:102419. Available from: https://doi.org/10.1016/j.jisa.2019.102419
[37] Kumar S, Gupta S, Arora S. Research trends in network-based intrusion detection systems: A review. IEEE Access. 2021;9:157761-157779. Available from: https://doi.org/10.1109/ACCESS.2021.3129775
[38] Kilincer IF, Ertam F, Sengur A. Machine learning methods for cyber security intrusion detection: Datasets and comparative study. Computer Networks. 2021;188:107840. Available from: https://doi.org/10.1016/j.comnet.2021.107840
[39] Jullian O, Otero B, Rodriguez E, Gutierrez N, Antona H, Canal R. Deep-learning based detection for cyber-attacks in IoT networks: A distributed attack detection framework. Journal of Network and Systems Management. 2023;31(2):33. Available from: https://link.springer.com/article/10.1007/s10922-023-09722-7
[40] Hariharan S, Jerusha YA, Suganeshwari G, Ibrahim SS, Tupakula U, Varadharajan V. A hybrid deep learning model for network intrusion detection system using Seq2Seq and ConvLSTM-subnets. IEEE Access. 2025. Available from: https://doi.org/10.1109/ACCESS.2025.3541399
[41] Nakıp M, Gelenbe E. Online self-supervised deep learning for intrusion detection systems. IEEE Transactions on Information Forensics and Security. 2024;19:5668-5683. Available from: https://doi.org/10.1109/TIFS.2024.3402148
[42] Wahab OA. Intrusion detection in the IoT under data and concept drifts: Online deep learning approach. IEEE Internet of Things Journal. 2022;9(20):19706-19716. Available from: https://doi.org/10.1109/JIOT.2022.3167005
[43] Lakshminarayana DH, Philips J, Tabrizi N. A survey of intrusion detection techniques. Proceedings of the 2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA). 2019:1122-1129. Available from: https://doi.org/10.1109/ICMLA.2019.00187
[44] Khamphakdee N, Benjamas N, Saiyod S. Improving intrusion detection system based on Snort rules for network probe attack detection. Proceedings of the 2014 2nd International Conference on Information and Communication Technology (ICoICT). 2014:69-74. Available from: https://doi.org/10.1109/ICoICT.2014.6914042
[45] Salem A, Cherubin G, Evans D, Köpf B, Paverd A, Suri A, Tople S, Zanella-Béguelin S. SoK: Let the privacy games begin! A unified treatment of data inference privacy in machine learning. Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP). 2023:327-345. Available from: https://doi.org/10.1109/SP46215.2023.10179281
[46] Bendiab G, Shiaeles S, Alruban A, Kolokotronis N. IoT malware network traffic classification using visual representation and deep learning. Proceedings of the 2020 6th IEEE Conference on Network Softwarization (NetSoft). 2020:444-449. Available from: https://doi.org/10.1109/NetSoft48620.2020.9165381
[47] Sun Y, Ochiai H, Esaki H. Decentralized deep learning for multi-access edge computing: A survey on communication efficiency and trustworthiness. IEEE Transactions on Artificial Intelligence. 2022;3(6):963-972. Available from: https://doi.org/10.1109/TAI.2021.3133819
[48] Manjunatha BA, Shastry KA, Naresh E, Pareek PK, Reddy KT. A network intrusion detection framework on sparse deep denoising auto-encoder for dimensionality reduction. Soft Computing. 2024;28(5):4503-4517. Available from: https://link.springer.com/article/10.1007/s00500-023-09408-x
[49] Siddique K, Akhtar Z, Khan FA, Kim Y. KDD Cup 99 data sets: A perspective on the role of data sets in network intrusion detection research. Computer. 2019;52(2):41-51. Available from: https://doi.org/10.1109/MC.2018.2888764
[50] Fidel G, Bitton R, Shabtai A. When explainability meets adversarial learning: Detecting adversarial examples using SHAP signatures. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN). 2020:1-8. Available from: https://doi.org/10.1109/IJCNN48605.2020.9207637
[51] Pothireddy SR. Cloud-native AI-driven enterprise automation for scalable digital process transformation in multi-industry ecosystems. Journal of Applied Sciences and Modelling. 2025;1(1):60-74. Available from: https://doi.org/10.71426/jasm.v1.i1.pp60-74
[52] Oyinna B, Udo PD, Nurhidayat I, Muslimyar AR. Integrating data processing and advanced analytics for scalable knowledge discovery in complex data environments. Journal of Computing and Data Technology. 2025;1(2):115-120. Available from: https://doi.org/10.71426/jcdt.v1.i2.pp115-120
[53] Penaganti R. Security-trust-determinism co-design using hybrid intrusion detection with temporal modeling for real-time publish-subscribe middleware. IEEE Communications Standards Magazine. 2026. Available from: https://doi.org/10.1109/MCOMSTD.2026.3676622
[54] Penaganti R. Graph neural network-based framework for real-time financial fraud detection in digital payment ecosystems. Journal of Computing and Data Technology. 2025;1(2):91-97. Available from: https://doi.org/10.71426/jcdt.v1.i2.pp91-97
[55] Safaei Yaraziz M, Jalili A, Gheisari M, Liu Y. Recent trends towards privacy-preservation in Internet of Things, its challenges and future directions. IET Circuits, Devices & Systems. 2023;17(2):53-61. Available from: https://doi.org/10.1049/cds2.12138
[56] Gheisari M, Ebrahimzadeh F, Rahimi M, Moazzamigodarzi M, Liu Y, Dutta Pramanik PK, Heravi MA, Mehbodniya A, Ghaderzadeh M, Feylizadeh MR, Kosari S. Deep learning: Applications, architectures, models, tools, and frameworks: A comprehensive survey. CAAI Transactions on Intelligence Technology. 2023;8(3):581-606. Available from: https://doi.org/10.1049/cit2.12180
[57] Lee CC, Lin TH, Tsai CS. A new authenticated group key agreement in a mobile environment. Annals of Telecommunications. 2009;64(11):735-744. Available from: https://doi.org/10.1007/s12243-009-0096-z
[58] Li CT, Lee CC, Weng CY. An extended chaotic maps based user authentication and privacy preserving scheme against DoS attacks in pervasive and ubiquitous computing environments. Nonlinear Dynamics. 2013;74(4):1133-1143. Available from: https://doi.org/10.1007/s11071-013-1029-y
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Mehdi Gheisari, Zhou Pingmei, Basheer Riskhan, Malusi Sibiya, Muhammad Faizan Khan, Seyed Kazem Gheblezadeh (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
