A Leakage-Controlled Evaluation Protocol for Machine-Learning Intrusion Detection on the TON_IoT Network Dataset
DOI:
https://doi.org/10.71426/jcdt.v2.i2.pp207-215Keywords:
Data leakage, Internet of Things (IoT), Intrusion detection, Shortcut learning.Abstract
Intrusion detectors trained on the TON_IoT network data routinely report near-perfect scores, but the extent to which these scores reflect attack behaviour rather than dataset-specific collection artefacts remains unclear. This study evaluates four classifiers using four progressively stricter evaluation protocols on the 211,043-flow train–test file: naive random splitting with all columns, removal of addressing and free-text identifiers, deduplication before splitting, and holding out entire attacker hosts. A probe trained only on IP addresses and ports reached a binary macro-F1 of 0.9999 and a ten-class macro-F1 of up to 0.932, indicating that the identifiers alone are highly predictive of the labels in this dataset. Exact duplicates made up 9.7% of rows, and 12.1% of test rows under naive cross-validation had a copy in the training fold. Removing identifiers cut multiclass macro-F1 from 0.986 to 0.953 for random forest, with man-in-the-middle F1 falling from 0.903 to 0.757. Under host hold-out, attack recall stayed above 0.95, while false-alarm rates on unseen benign hosts varied substantially across hold-outs. The evaluation protocols, identifier-only probe, and reporting checklist provide practical safeguards for future studies using this dataset.
References
[1] Heidari A, Jabraeil Jamali MA. Internet of Things intrusion detection systems: a comprehensive review and future directions. Cluster Computing. 2023;26(6):3753–3780. Available from: https://doi.org/10.1007/s10586-022-03776-z
[2] Gyamfi E, Jurcut A. Intrusion Detection in Internet of Things Systems: A Review on Design Approaches Leveraging Multi-Access Edge Computing, Machine Learning, and Datasets. Sensors. 2022;22(10):3744. Available from: https://doi.org/10.3390/s22103744
[3] Haque S, El-Moussa F, Komninos N, Rajarajan M. A Systematic Review of Data-Driven Attack Detection Trends in IoT. Sensors. 2023;23(16):7191. Available from: https://doi.org/10.3390/s23167191
[4] Moustafa N. A new distributed architecture for evaluating AI-based security systems at the edge: Network TON_IoT datasets. Sustainable Cities and Society. 2021;72:102994. Available from: https://doi.org/10.1016/j.scs.2021.102994
[5] Alsaedi A, Moustafa N, Tari Z, Mahmood A, Anwar A. TON_IoT Telemetry Dataset: A New Generation Dataset of IoT and IIoT for Data-Driven Intrusion Detection Systems. IEEE Access. 2020;8:165130–165150. Available from: https://doi.org/10.1109/ACCESS.2020.3022862
[6] Booij TM, Chiscop I, Meeuwissen E, Moustafa N, den Hartog FTH. ToN_IoT: The Role of Heterogeneity and the Need for Standardization of Features and Attack Types in IoT Network Intrusion Data Sets. IEEE Internet of Things Journal. 2022;9(1):485–496. Available from: https://doi.org/10.1109/JIOT.2021.3085194
[7] Gad AR, Nashat AA, Barkat TM. Intrusion Detection System Using Machine Learning for Vehicular Ad Hoc Networks Based on ToN-IoT Dataset. IEEE Access. 2021;9:142206–142217. Available from: https://doi.org/10.1109/ACCESS.2021.3120626
[8] Lo WW, Layeghy S, Sarhan M, Gallagher M, Portmann M. E-GraphSAGE: A Graph Neural Network based Intrusion Detection System for IoT. In: Proceedings of the NOMS 2022–2022 IEEE/IFIP Network Operations and Management Symposium. IEEE; 2022. p. 1–9. Available from: https://doi.org/10.1109/NOMS54207.2022.9789878
[9] Sarhan M, Layeghy S, Moustafa N, Gallagher M, Portmann M. Feature extraction for machine learning-based intrusion detection in IoT networks. Digital Communications and Networks. 2024;10(1):205–216. Available from: https://doi.org/10.1016/j.dcan.2022.08.012
[10] Arp D, Quiring E, Pendlebury F, Warnecke A, Pierazzi F, Wressnegger C, et al. Pitfalls in Machine Learning for Computer Security. Communications of the ACM. 2024;67(11):104–112. Available from: https://doi.org/10.1145/3643456
[11] Apruzzese G, Laskov P, Schneider J. SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection. In: Proceedings of the 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE; 2023. p. 592–614. Available from: https://doi.org/10.1109/EuroSP57164.2023.00042
[12] Engelen G, Rimmer V, Joosen W. Troubleshooting an Intrusion Detection Dataset: The CICIDS2017 Case Study. In: Proceedings of the 2021 IEEE Security and Privacy Workshops (SPW). IEEE; 2021. p. 7–12. Available from: https://doi.org/10.1109/SPW53761.2021.00009
[13] Lanvin M, Gimenez P-F, Han Y, Majorczyk F, Mé L, Totel É. Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes. In: Risks and Security of Internet and Systems. Lecture Notes in Computer Science, vol. 13857. Cham: Springer; 2023. p. 18–33. Available from: https://doi.org/10.1007/978-3-031-31108-6_2
[14] Kaufman S, Rosset S, Perlich C, Stitelman O. Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data. 2012;6(4):1–21. Available from: https://doi.org/10.1145/2382577.2382579
[15] Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023;4(9):100804. Available from: https://doi.org/10.1016/j.patter.2023.100804
[16] D'hooge L, Verkerken M, Volckaert B, Wauters T, De Turck F. Establishing the Contaminating Effect of Metadata Feature Inclusion in Machine-Learned Network Intrusion Detection Models. In: Detection of Intrusions and Malware, and Vulnerability Assessment. Lecture Notes in Computer Science, vol. 13358. Cham: Springer; 2022. p. 23–41. Available from: https://doi.org/10.1007/978-3-031-09484-2_2
[17] Sarhan M, Layeghy S, Portmann M. Towards a Standard Feature Set for Network Intrusion Detection System Datasets. Mobile Networks and Applications. 2022;27(1):357–370. Available from: https://doi.org/10.1007/s11036-021-01843-0
[18] Sarhan M, Layeghy S, Portmann M. Evaluating Standard Feature Sets Towards Increased Generalisability and Explainability of ML-Based Network Intrusion Detection. Big Data Research. 2022;30:100359. Available from: https://doi.org/10.1016/j.bdr.2022.100359
[19] Al-Hawawreh M, Sitnikova E, Aboutorab N. X-IIoTID: A Connectivity-Agnostic and Device-Agnostic Intrusion Data Set for Industrial Internet of Things. IEEE Internet of Things Journal. 2022;9(5):3962–3977. Available from: https://doi.org/10.1109/JIOT.2021.3102056
[20] Ferrag MA, Friha O, Hamouda D, Maglaras L, Janicke H. Edge-IIoTset: A New Comprehensive Realistic Cyber Security Dataset of IoT and IIoT Applications for Centralized and Federated Learning. IEEE Access. 2022;10:40281–40306. Available from: https://doi.org/10.1109/ACCESS.2022.3165809
[21] Neto ECP, Dadkhah S, Ferreira R, Zohourian A, Lu R, Ghorbani AA. CICIoT2023: A Real-Time Dataset and Benchmark for Large-Scale Attacks in IoT Environment. Sensors. 2023;23(13):5941. Available from: https://doi.org/10.3390/s23135941
[22] Layeghy S, Portmann M. Explainable cross-domain evaluation of ML-based network intrusion detection systems. Computers & Electrical Engineering. 2023;108:108692. Available from: https://doi.org/10.1016/j.compeleceng.2023.108692
[23] Alani MM, Miri A. Towards an Explainable Universal Feature Set for IoT Intrusion Detection. Sensors. 2022;22(15):5690. Available from: https://doi.org/10.3390/s22155690
[24] Friha O, Ferrag MA, Shu L, Maglaras L, Choo K-KR, Nafaa M. FELIDS: Federated learning-based intrusion detection system for agricultural Internet of Things. Journal of Parallel and Distributed Computing. 2022;165:17–31. Available from: https://doi.org/10.1016/j.jpdc.2022.03.003
[25] Altunay HC, Albayrak Z. A hybrid CNN+LSTM-based intrusion detection system for industrial IoT networks. Engineering Science and Technology, an International Journal. 2023;38:101322. Available from: https://doi.org/10.1016/j.jestch.2022.101322
[26] Saba T, Rehman A, Sadad T, Kolivand H, Bahaj SA. Anomaly-based intrusion detection system for IoT networks through deep learning model. Computers & Electrical Engineering. 2022;99:107810. Available from: https://doi.org/10.1016/j.compeleceng.2022.107810
[27] Naz N, Khan MA, Alsuhibany SA, Diyan M, Tan Z, Khan MA, et al. Ensemble learning-based IDS for sensors telemetry data in IoT networks. Mathematical Biosciences and Engineering. 2022;19(10):10550–10580. Available from: https://doi.org/10.3934/mbe.2022493
[28] Liu L, Engelen G, Lynar T, Essam D, Joosen W. Error Prevalence in NIDS Datasets: A Case Study on CIC-IDS-2017 and CSE-CIC-IDS-2018. In: Proceedings of the 2022 IEEE Conference on Communications and Network Security (CNS). IEEE; 2022. p. 254–262. Available from: https://doi.org/10.1109/CNS56114.2022.9947235
[29] Guerra JL, Catania C, Veas E. Datasets are not enough: Challenges in labeling network traffic. Computers & Security. 2022;120:102810. Available from: https://doi.org/10.1016/j.cose.2022.102810
[30] Apruzzese G, Laskov P, Montes de Oca E, Mallouli W, Brdalo Rapa L, Grammatopoulos AV, et al. The Role of Machine Learning in Cybersecurity. Digital Threats: Research and Practice. 2023;4(1):1–38. Available from: https://doi.org/10.1145/3545574
[31] Andresini G, Pendlebury F, Pierazzi F, Loglisci C, Appice A, Cavallaro L. INSOMNIA: Towards Concept-Drift Robustness in Network Intrusion Detection. In: Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security. ACM; 2021. p. 111–122. Available from: https://doi.org/10.1145/3474369.3486864
[32] Whalen S, Schreiber J, Noble WS, Pollard KS. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics. 2022;23(3):169–181. Available from: https://doi.org/10.1038/s41576-021-00434-9
[33] Rosenblatt M, Tejavibulya L, Jiang R, Noble WS, Scheinost D. Data leakage inflates prediction performance in connectome-based machine learning models. Nature Communications. 2024;15(1):1829. Available from: https://doi.org/10.1038/s41467-024-46150-w
[34] Geirhos R, Jacobsen J-H, Michaelis C, Zemel R, Brendel W, Bethge M, et al. Shortcut learning in deep neural networks. Nature Machine Intelligence. 2020;2(11):665–673. Available from: https://doi.org/10.1038/s42256-020-00257-z
[35] Verkerken M, D'hooge L, Wauters T, Volckaert B, De Turck F. Towards Model Generalization for Intrusion Detection: Unsupervised Machine Learning Techniques. Journal of Network and Systems Management. 2022;30(1):12. Available from: https://doi.org/10.1007/s10922-021-09615-7
[36] Hung R-J. An Explainable XGBoost-Based Framework for IoT Attack Detection with Unseen Attack Family Evaluation. Sensors. 2026;26(10):3005. Available from: https://doi.org/10.3390/s26103005
[37] Moustafa N. ToN_IoT Datasets [dataset]. IEEE DataPort. 2019 Oct 16. Available from: https://doi.org/10.21227/fesz-dm97
[38] Shwartz-Ziv R, Armon A. Tabular data: Deep learning is not all you need. Information Fusion. 2022;81:84–90. Available from: https://doi.org/10.1016/j.inffus.2021.11.011
[39] Neupane S, Ables J, Anderson W, Mittal S, Rahimi S, Banicescu I, et al. Explainable Intrusion Detection Systems (X-IDS): A Survey of Current Methods, Challenges, and Opportunities. IEEE Access. 2022;10:112392–112415. Available from: https://doi.org/10.1109/ACCESS.2022.3216617
[40] Moustafa N, Koroniotis N, Keshk M, Zomaya AY, Tari Z. Explainable Intrusion Detection for Cyber Defences in the Internet of Things: Opportunities and Solutions. IEEE Communications Surveys & Tutorials. 2023;25(3):1775–1807. Available from: https://doi.org/10.1109/COMST.2023.3280465
[41] Le T-T-H, Kim H, Kang H, Kim H. Classification and Explanation for Intrusion Detection System Based on Ensemble Trees and SHAP Method. Sensors. 2022;22(3):1154. Available from: https://doi.org/10.3390/s22031154
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Reginald Ekene Ogu (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors retain copyright in their published work subject to the journal's publication and licensing arrangements.
Articles published in JMT are distributed under the Creative Commons Attribution–NonCommercial 4.0 International License (CC BY-NC 4.0).
Under this license, readers may share and adapt published material for non-commercial purposes, provided that:
- Appropriate credit is given to the author or authors.
- The original publication source is acknowledged.
- A link to the applicable license is provided.
- Any modifications are clearly indicated.
Commercial reuse is not permitted under CC BY-NC 4.0 without appropriate authorization from the relevant rights holder.
License details are available at:
Creative Commons Attribution–NonCommercial 4.0 International
DOI prefix: 10.71426
