Integrating Data Processing and Advanced Analytics for Scalable Knowledge Discovery in Complex Data Environments
DOI:
https://doi.org/10.71426/jcdt.v1.i2.pp115-120Keywords:
Data Processing, Big data analytics, Machine learning, Knowledge discovery, Intelligent applicationsAbstract
The rapid proliferation of heterogeneous and high-dimensional data across domains has amplified the demand for efficient data processing and robust analytical frameworks. This paper presents a comprehensive study on integrating data processing and analysis as a unified paradigm to enable effective knowledge discovery and intelligent applications. Data processing encompasses systematic techniques for acquisition, cleaning, transformation, and integration of raw data into consistent and reliable forms. Coupled with advanced analytical approaches, including statistical modeling, machine learning, and deep learning, these processes collectively transform unstructured information into actionable insights. The proposed perspective emphasizes scalable pipelines, real-time processing frameworks, and outlier-robust mechanisms to ensure reliability across large and dynamic datasets. Furthermore, the integration of descriptive, predictive, and prescriptive analytics demonstrates the potential for enhanced decision-making in critical sectors such as healthcare, energy systems, finance, and governance. The paper also highlights emerging challenges, including interpretability, privacy preservation, and ethical considerations, while underscoring future research opportunities in quantum data analysis and federated learning. By bridging data processing and analysis, this study advocates a holistic approach that fosters transparent, adaptive, and scalable knowledge discovery, ultimately strengthening the role of data-driven intelligence in addressing complex real-world problems.
References
[1] Fragkoulis M, Carbone P, Kalavri V, Katsifodimos A. A survey on the evolution of stream processing systems. The VLDB Journal. 2024;33(2):507–541. Available from: https://doi.org/10.1007/s00778-023-00819-8
[2] Demirezen MU, Navruz TS. Performance analysis of lambda architecture-based big-data systems on air/ground surveillance application with ADS-B data. Sensors. 2023;23(17):7580. Available from: https://doi.org/10.3390/s23177580
[3] Šprem Š, Tomažin N, Matečić J, Horvat M. Building advanced web applications using data ingestion and data processing tools. Electronics. 2024;13(4):709. Available from: https://doi.org/10.3390/electronics13040709
[4] Lin J. The lambda and the kappa. IEEE Internet Computing. 2017;21(5):60–66. Available from: https://ieeexplore.ieee.org/document/8039313
[5] Yaghi MK, Haji M, Thaher M, Kassem I. Big data pipeline: An overview of ingestion and preparation tools. American Academic & Scholarly Research Journal. 2025;14(13). Available from: https://aasrc.org/aasrj/index.php/aasrj/article/view/2324/0
[6] Jäger S, Biessmann F. From data imputation to data cleaning—automated cleaning of tabular data improves downstream predictive performance. In: International Conference on Artificial Intelligence and Statistics (AISTATS); 2024 Apr 18; pp. 3394–3402. PMLR. Available from: https://proceedings.mlr.press/v238/jager24a
[7] Zhou Y, Aryal S, Bouadjenek MR. Review for handling missing data with special missing mechanism. arXiv preprint. 2024;arXiv:2404.04905. Available from: https://doi.org/10.48550/arXiv.2404.04905
[8] Afkanpour M, Hosseinzadeh E, Tabesh H. Identify the most appropriate imputation method for handling missing values in clinical structured datasets: A systematic review. BMC Medical Research Methodology. 2024;24(1):188. Available from: https://doi.org/10.1186/s12874-024-02310-6
[9] Zamanzadeh Darban Z, Webb GI, Pan S, Aggarwal C, Salehi M. Deep learning for time series anomaly detection: A survey. ACM Computing Surveys. 2024;57(1):1–42. Available from: https://doi.org/10.1145/3691338
[10] Wang F, Jiang Y, Zhang R, Wei A, Xie J, Pang X. A survey of deep anomaly detection in multivariate time series: Taxonomy, applications, and directions. Sensors. 2025;25(1):190. Available from: https://doi.org/10.3390/s25010190
[11] Guo M, Wang Y, Yang Q, Li R, Zhao Y, Li C, Zhu M, Cui Y, Jiang X, Sheng S, Li Q. Normal workflow and key strategies for data cleaning toward real-world data. Interactive Journal of Medical Research. 2023;12(1):e44310. Available from: https://doi.org/10.2196/44310
[12] Borrohou S, Fissoune R, Badir H. Data cleaning survey and challenges—improving outlier detection algorithm in machine learning. Journal of Smart Cities and Society. 2023;2(3):125–140. Available from: https://doi.org/10.3233/SCS-230008
[13] Guha S, Khan FA, Stoyanovich J, Schelter S. Automated data cleaning can hurt fairness in machine learning-based decision making. IEEE Transactions on Knowledge and Data Engineering. 2024;36(12):7368–7379. Available from: https://doi.org/10.1109/TKDE.2024.3365524
[14] Peng C, Xia F, Naseriparsa M, Osborne F. Knowledge graphs: Opportunities and challenges. Artificial Intelligence Review. 2023;56(11):13071–13102. Available from: https://doi.org/10.1007/s10462-023-10465-9
[15] Niu G. Knowledge graph embeddings: A comprehensive survey on capturing relation properties. arXiv preprint. 2024;arXiv:2410.14733. Available from: https://doi.org/10.48550/arXiv.2410.14733
[16] Zhang Y, Floratou A, Cahoon J, Krishnan S, Müller AC, Banda D, Psallidas F, Patel JM. Schema matching using pre-trained language models. In: 2023 IEEE 39th International Conference on Data Engineering (ICDE); 2023 Apr 3; pp. 1558–1571. IEEE. Available from: https://ieeexplore.ieee.org/document/10184612
[17] Liu Y, Pena E, Santos A, Wu E, Freire J. Magneto: Combining small and large language models for schema matching. arXiv preprint. 2024;arXiv:2412.08194. Available from: https://doi.org/10.48550/arXiv.2412.08194
[18] Ceravolo P, Azzini A, Angelini M, Catarci T, Cudré-Mauroux P, Damiani E, Mazak A, Van Keulen M, Jarrar M, Santucci G, Sattler KU. Big data semantics. Journal on Data Semantics. 2018;7(2):65–85. Available from: https://doi.org/10.1007/s13740-018-0086-2
[19] Ramonell C, Chacón R, Posada H. Knowledge graph-based data integration system for digital twins of built assets. Automation in Construction. 2023;156:105109. Available from: https://doi.org/10.1016/j.autcon.2023.105109
[20] Correia L, Goos JC, Klein P, Bäck T, Kononova AV. Online model-based anomaly detection in multivariate time series: Taxonomy, survey, research challenges and future directions. Engineering Applications of Artificial Intelligence. 2024;138:109323. Available from: https://doi.org/10.1016/j.engappai.2024.109323
[21] Cesario E. Big data analytics and smart cities: Applications, challenges, and opportunities. Frontiers in Big Data. 2023;6:1149402. Available from: https://doi.org/10.3389/fdata.2023.1149402
[22] Osman AMS. A novel big data analytics framework for smart cities. Future Generation Computer Systems. 2019;91:620–633. Available from: https://doi.org/10.1016/j.future.2018.06.046
[23] Hai R, Koutras C, Quix C, Jarke M. Data lakes: A survey of functions and systems. IEEE Transactions on Knowledge and Data Engineering. 2023;35(12):12571–12590. Available from: https://doi.org/10.1109/TKDE.2023.3270101
[24] Harby AA, Zulkernine F. Data lakehouse: A survey and experimental study. Information Systems. 2025;127:102460. Available from: https://doi.org/10.1016/j.is.2024.102460
[25] Noor M, Baharom F, Mohd H. Big data governance framework: Current and future trends. Journal of Research and Digital Innovation. 2025;1(1):26–36. Available from: http://103.250.10.42/index.php/RDI/article/view/5/4
[26] Acev D, Biyani S, Rieder F, Aldenhoff TT, Blazevic M, Riehle DM, Wimmer MA. Systematic analysis of data governance frameworks and their relevance to data trusts. Management Review Quarterly. 2025:1–54. Available from: https://doi.org/10.1007/s11301-025-00545-1
[27] Goldstein I, Spatt CS, Ye M. Big data in finance. Review of Financial Studies. 2021;34(7):3213–3225. Available from: https://doi.org/10.1093/rfs/hhab038
[28] Karimi Y, Haghi Kashani M, Akbari M, Mahdipour E. Leveraging big data in smart cities: A systematic review. Concurrency and Computation: Practice and Experience. 2021;33(21):e6379. Available from: https://doi.org/10.1002/cpe.6379
[29] Warren J, Marz N. Big data: Principles and best practices of scalable real-time data systems. Simon and Schuster. 2015. Available from: https://books.google.co.in/books?id=XjszEAAAQBAJ&pg=PT14
[30] Abedjan Z, Golab L, Naumann F. Profiling relational data: A survey. The VLDB Journal. 2015;24(4):557–581. Available from: https://doi.org/10.1007/s00778-015-0389-y
[31] Alareeni B. Big data in finance: Transforming the financial landscape. Studies in Big Data. 2025;Volume 1. Available from: https://doi.org/10.1007/978-3-031-75095-3
[32] Saripudi K. A study on artificial intelligence and cloud computing assistance for enhancement of startup businesses. Journal of Computing and Data Technology. 2025;1(1):68–76. Available from: https://doi.org/10.71426/jcdt.v1.i1.pp68-76
[33] Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A. A survey on bias and fairness in machine learning. ACM Computing Surveys. 2021;54(6):1–35. Available from: https://doi.org/10.1145/3457607
[34] Begenau J, Farboodi M, Veldkamp L. Big data in finance and the growth of large firms. Journal of Monetary Economics. 2018;97:71–87. Available from: https://doi.org/10.1016/j.jmoneco.2018.05.013
[35] Liu J, Fu S. Financial big data management and intelligence based on computer intelligent algorithm. Scientific Reports. 2024;14(1):9395. Available from: https://doi.org/10.1038/s41598-024-59244-8
[36] Heath T, Bizer C. Linked data. Synthesis Lectures on Data, Semantics and Knowledge. 2011. Available from: https://doi.org/10.1007/978-3-031-79432-2
[37] Lim B, Arık SÖ, Loeff N, Pfister T. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting. 2021;37(4):1748–1764. Available from: https://doi.org/10.1016/j.ijforecast.2021.03.012
[38] Borisov V, Leemann T, Seßler K, Haug J, Pawelczyk M, Kasneci G. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems. 2024;35(6):7499–7519. Available from: https://doi.org/10.1109/TNNLS.2022.3229161
[39] Dietterich TG. Ensemble methods in machine learning. In: International Workshop on Multiple Classifier Systems; 2000 Jun 21; pp. 1–15. Berlin: Springer. Available from: https://doi.org/10.1007/3-540-45014-9_1
[40] Gama J, Žliobaitė I, Bifet A, Pechenizkiy M, Bouchachia A. A survey on concept drift adaptation. ACM Computing Surveys. 2014;46(4):1–37. Available from: https://doi.org/10.1145/2523813
[41] McMahan HB, Moore E, Ramage D, Hampson S, Arcas BA. Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS); 2017; Vol. 54. Available from: https://proceedings.mlr.press/v54/mcmahan17a/mcmahan17a.pdf
[42] Tao F, Zhang H, Liu A, Nee AY. Digital twin in industry: State-of-the-art. IEEE Transactions on Industrial Informatics. 2019;15(4):2405–2415. Available from: https://doi.org/10.1109/TII.2018.2873186
[43] Karatas M, Eriskin L, Deveci M, Pamucar D, Garg H. Big data for healthcare Industry 4.0: Applications, challenges and future perspectives. Expert Systems with Applications. 2022;200:116912. Available from: https://doi.org/10.1016/j.eswa.2022.116912
[44] Bahri S, Zoghlami N, Abed M, Tavares JM. Big data for healthcare: A survey. IEEE Access. 2019;7:7397–7408. Available from: https://ieeexplore.ieee.org/abstract/document/8585021
[45] Wu Z, Pan S, Chen F, Long G, Zhang C, Yu PS. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems. 2021;32(1):4–24. Available from: https://doi.org/10.1109/TNNLS.2020.2978386
[46] Rajesh M, Ramachandran S, Vengatesan K, Dhanabalan SS, Nataraj SK. Federated learning for personalized recommendation in securing power traces in smart grid systems. IEEE Transactions on Consumer Electronics. 2024;70(1):88–95. Available from: https://doi.org/10.1109/TCE.2024.3368087
[47] Judge MA, Franzitta V, Curto D, Guercio A, Cirrincione G, Khattak HA. A comprehensive review of artificial intelligence approaches for smart grid integration and optimization. Energy Conversion and Management X. 2024;24:100724. Available from: https://doi.org/10.1016/j.ecmx.2024.100724
[48] Mousavi SAE, Chabanloo RM, Farrokhifar M, Pozo D. Wide area backup protection scheme for distance relays considering the uncertainty of network protection. Electric Power Systems Research. 2020;189:106651. Available from: https://doi.org/10.1016/j.epsr.2020.106651
[49] Zabihi A, Parhamfar M, Khodadadi M. Strengthening resilience: A brief review of cybersecurity challenges in IoT-driven smart grids. Journal of Modern Technology. 2024;1(2):106–120. Available from: https://doi.org/10.71426/jmt.v1.i2.pp106-120
[50] Pagidela Y, N V. A short review on optimal allocation of microgrid. Journal of Modern Technology. 2024;1(2):132–140. Available from: https://doi.org/10.71426/jmt.v1.i2.pp132-140
[51] Li Y, Yu C, Shahidehpour M, Yang T, Zeng Z, Chai T. Deep reinforcement learning for smart grid operations: Algorithms, applications, and prospects. Proceedings of the IEEE. 2023;111(9):1055–1096. Available from: https://doi.org/10.1109/JPROC.2023.3303358
[52] Santhakumar S, Meerman H, Faaij A. Improving the analytical framework for quantifying technological progress in energy technologies. Renewable and Sustainable Energy Reviews. 2021;145:111084. Available from: https://doi.org/10.1016/j.rser.2021.111084
[53] Soma AK. Hybrid RNN-GRU-LSTM model for accurate detection of DDOS attacks on IDS dataset. Journal of Modern Technology. 2024;2(1):283–291. Available from: https://doi.org/10.71426/jmt.v2.i1.pp283-291
[54] Kara RV. SmartBio: An AI-enabled smart medical device for early cancer detection using variational autoencoders and multimodal sensor integration. Journal of Modern Technology. 2025;2(1):292–301. Available from: https://doi.org/10.71426/jmt.v2.i1.pp292-301
[55] Penaganti R. AI-driven fraud detection in financial systems: A technical deep dive. Journal of Information Systems Engineering & Management. 2025;10(60S):1049–1069. Available from: https://doi.org/10.52783/jisem.v10i60s.13262
[56] Sitharthan R, Vimal S, Verma A, Karthikeyan M, Dhanabalan SS, Prabaharan N, et al. Smart microgrid with the internet of things for adequate energy management and analysis. Computers & Electrical Engineering. 2022;106:108556. Available from: https://doi.org/10.1016/j.compeleceng.2022.108556
Downloads
Issue
Section
License
Copyright (c) 2025 Benneth Oyinna, Peter David Udo, Irfan Nurhidayat, Abdul Raqib Muslimyar (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
