A Systematic Review on Hybrid Deep Learning and Metaheuristic Optimization Techniques for Medical Image Segmentation and Classification
DOI:
https://doi.org/10.71426/jcdt.v2.i1.pp151-164Keywords:
Medical image segmentation, Medical image classification, Deep learning, Metaheuristic optimization, Artificial Intelligence (AI), Vision transformers, Explainable Artificial Intelligence (XAI).Abstract
Medical image segmentation and classification have become fundamental components of modern intelligent healthcare systems due to their ability to support early disease diagnosis, treatment planning, prognosis evaluation, and computer-aided clinical decision-making. Recent advances in deep learning have significantly improved the capability of automated medical image analysis systems across multiple imaging modalities including X-ray, computed tomography, magnetic resonance imaging, ultrasound, histopathology, and retinal imaging. However, despite remarkable performance improvements, conventional deep learning models still face several limitations related to hyperparameter tuning, computational complexity, convergence instability, local minima stagnation, class imbalance, limited interpretability, and poor generalization under heterogeneous clinical conditions. To address these challenges, hybrid frameworks integrating deep learning with metaheuristic optimization techniques have emerged as a promising research direction for improving segmentation accuracy, classification robustness, feature optimization, and computational efficiency. This review article covers the historical evolution of intelligent medical image analysis, theoretical foundations of optimization-driven learning, taxonomy of deep learning architectures, and the role of evolutionary and swarm-based optimization algorithms including genetic algorithms, particle swarm optimization, grey wolf optimizer, firefly algorithm, whale optimization, ant colony optimization, Bayesian optimization, and reptile search optimization. Comparative analysis of datasets, evaluation metrics, computational complexity, convergence behavior, and clinical deployment challenges is also presented. Finally, open research challenges and future directions are identified toward trustworthy, interpretable, scalable, and autonomous AI-driven medical imaging systems for next-generation intelligent healthcare applications.
References
[1] Litjens G, Kooi T, Bejnordi BE, Setio AA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Ginneken B, Sánchez CI. A survey on deep learning in medical image analysis. Medical Image Analysis. 2017;42:60-88. Available from: https://doi.org/10.1016/j.media.2017.07.005
[2] Shen D, Wu G, Suk HI. Deep Learning in Medical Image Analysis. Annual Review of Biomedical Engineering. 2017;19:221-248. Available from: https://doi.org/10.1146/annurev-bioeng-071516-044442
[3] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. Lecture Notes in Computer Science. 2015;9351:234-241. Available from: https://doi.org/10.1007/978-3-319-24574-4_28
[4] Long J, Shelhamer E, Darrell T. Fully Convolutional Networks for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017;39(4):640-651. Available from: https://doi.org/10.1109/TPAMI.2016.2572683
[5] Milletari F, Navab N, Ahmadi SA. V-Net: Fully convolutional Neural Networks for Volumetric Medical Image Segmentation. 2016 Fourth International Conference on 3D Vision. 2016:565-571. Available from: https://doi.org/10.1109/3DV.2016.79
[6] Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J. UNet++: A nested U-Net architecture for medical image segmentation. Lecture Notes in Computer Science. 2018;11045:3-11. Available from: https://doi.org/10.1007/978-3-030-00889-5_1
[7] Schlemper J, Oktay O, Schaap M, Heinrich M, Kainz B, Glocker B, Rueckert D. Attention gated networks: Learning to leverage salient regions in medical images. Medical Image Analysis. 2019;53:197-207. Available from: https://doi.org/10.1016/j.media.2019.01.012
[8] He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:770-778. Available from: https://doi.org/10.1109/CVPR.2016.90
[9] Huang G, Liu Z, van der Maaten L, Weinberger KQ. Densely connected convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:4700-4708. Available from: https://doi.org/10.1109/CVPR.2017.243
[10] Badrinarayanan V, Kendall A, Cipolla R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017;39(12):2481-2495. Available from: https://doi.org/10.1109/TPAMI.2016.2644615
[11] Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2018;40(4):834-848. Available from: https://doi.org/10.1109/TPAMI.2017.2699184
[12] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I. Attention is All You Need. Advances in Neural Information Processing Systems. 2017;30:5998-6008. Available from: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
[13] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations. 2021. Available from: https://arxiv.org/abs/2010.11929
[14] Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Transactions on Medical Imaging. 2016;35(5):1299-1312. Available from: https://doi.org/10.1109/TMI.2016.2535302
[15] Yamashita R, Nishio M, Do RKG, Togashi K. Convolutional neural networks: An overview and application in radiology. Insights into Imaging. 2018;9:611-629. Available from: https://doi.org/10.1007/s13244-018-0639-9
[16] Lundervold AS, Lundervold A. An overview of deep learning in medical imaging focusing on MRI. Zeitschrift für Medizinische Physik. 2019;29(2):102-127. Available from: https://doi.org/10.1016/j.zemedi.2018.11.002
[17] Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115-118. Available from: https://doi.org/10.1038/nature21056
[18] Ardila D, Kiraly AP, Bharadwaj S, Choi B, Reicher JJ, Peng L, Tse D, Etemadi M, Ye W, Corrado G, Naidich DP, Shetty S. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nature Medicine. 2019;25:954-961. Available from: https://doi.org/10.1038/s41591-019-0447-x
[19] Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin PM, Larochelle H. Brain tumor segmentation with Deep Neural Networks. Medical Image Analysis. 2017;35:18-31. Available from: https://doi.org/10.1016/j.media.2016.05.004
[20] Kennedy J, Eberhart R. Particle swarm optimization. Proceedings of ICNN'95 – International Conference on Neural Networks. 1995;4:1942-1948. Available from: https://doi.org/10.1109/ICNN.1995.488968
[21] Mirjalili S. Grey Wolf Optimizer. Advances in Engineering Software. 2014;69:46-61. Available from: https://doi.org/10.1016/j.advengsoft.2013.12.007
[22] Yang XS. Firefly Algorithms for Multimodal Optimization. Lecture Notes in Computer Science. 2009;5792:169-178. Available from: https://doi.org/10.1007/978-3-642-04944-6_14
[23] Mirjalili S, Lewis A. The Whale Optimization Algorithm. Advances in Engineering Software. 2016;95:51-67. Available from: https://doi.org/10.1016/j.advengsoft.2016.01.008
[24] Storn R, Price K. Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces. Journal of Global Optimization. 1997;11:341-359. Available from: https://doi.org/10.1023/A:1008202821328
[25] Bergstra J, Bardenet R, Bengio Y, Kégl B. Algorithms for Hyper-Parameter Optimization. Advances in Neural Information Processing Systems. 2011;24. Available from: https://proceedings.neurips.cc/paper_files/paper/2011/hash/86e8f7ab32cfd12577bc2619bc635690-Abstract.html
[26] Abdar M, Pourpanah F, Hussain S, Rezazadegan D, Liu L, Ghavamzadeh M, Fieguth P, Cao X, Khosravi A, Acharya UR, Makarenkov V. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion. 2021;76:243-297. Available from: https://doi.org/10.1016/j.inffus.2021.05.008
[27] McMahan B, Moore E, Ramage D, Hampson S, y Arcas BA. Communication-Efficient Learning of Deep Networks from Decentralized Data. Artificial Intelligence and Statistics. 2017. p. 1273-1282. Available from: https://proceedings.mlr.press/v54/mcmahan17a.html
[28] Touvron H, Cord M, Douze M, Massa F, Sablayrolles A, Jégou H. Training data-efficient image transformers & distillation through attention. International Conference on Machine Learning. 2021. p. 10347-10357. Available from: https://proceedings.mlr.press/v139/touvron21a.html
[29] Tan M, Le Q. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International Conference on Machine Learning. 2019. p. 6105-6114. Available from: https://proceedings.mlr.press/v97/tan19a.html
[30] Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, Glocker B. Attention U-Net: Learning Where to Look for the Pancreas. arXiv preprint arXiv:1804.03999. 2018. Available from: https://openreview.net/forum?id=Skft7cijM
[31] Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint arXiv:1704.04861. 2017. Available from: https://doi.org/10.48550/arXiv.1704.04861
[32] Kingma DP, Ba J. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980. 2014. Available from: https://doi.org/10.48550/arXiv.1412.6980
[33] Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. International Conference on Machine Learning. 2015. p. 448-456. Available from: https://proceedings.mlr.press/v37/ioffe15.html
[34] Akiba T, Sano S, Yanase T, Ohta T, Koyama M. Optuna: A Next-generation Hyperparameter Optimization Framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019. p. 2623-2631. Available from: https://doi.org/10.1145/3292500.3330701
[35] Goodfellow I, Bengio Y, Courville A, Bengio Y. Deep learning. MIT Press. 2016. Available from: https://www.deeplearningbook.org
[36] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998;86(11):2278-2324. Available from: https://doi.org/10.1109/5.726791
[37] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436-444. Available from: https://doi.org/10.1038/nature14539
[38] Bengio Y, Courville A, Vincent P. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2013;35(8):1798-1828. Available from: https://doi.org/10.1109/TPAMI.2013.50
[39] Miotto R, Wang F, Wang S, Jiang X, Dudley JT. Deep learning for healthcare: review, opportunities and challenges. Briefings in Bioinformatics. 2018;19(6):1236-1246. Available from: https://doi.org/10.1093/bib/bbx044
[40] Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine. 2019;25(1):44-56. Available from: https://doi.org/10.1038/s41591-018-0300-7
[41] Rajpurkar P, Irvin J, Zhu K, Yang B, Mehta H, Duan T, Ding D, Bagul A, Langlotz C, Shpanskaya K, Lungren MP. CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning. arXiv preprint arXiv:1711.05225. 2017. Available from: https://doi.org/10.48550/arXiv.1711.05225
[42] Suzuki K. Overview of deep learning in medical imaging. Radiological Physics and Technology. 2017;10(3):257-273. Available from: https://doi.org/10.1007/s12194-017-0406-5
[43] Greenspan H, Van Ginneken B, Summers RM. Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique. IEEE Transactions on Medical Imaging. 2016;35(5):1153-1159. Available from: https://doi.org/10.1109/TMI.2016.2553401
[44] Litjens G, Ciompi F, Wolterink JM, de Vos BD, Leiner T, Teuwen J, Išgum I. State-of-the-Art Deep Learning in Cardiovascular Image Analysis. JACC: Cardiovascular Imaging. 2019;12(8 Part 1):1549-1565. Available from: https://doi.org/10.1016/j.jcmg.2019.06.009
[45] Raghu M, Zhang C, Kleinberg J, Bengio S. Transfusion: Understanding Transfer Learning for Medical Imaging. Advances in Neural Information Processing Systems. 2019;32. Available from: https://proceedings.neurips.cc/paper/2019/hash/eb1e78328c46506b46a4ac4a1e378b91-Abstract.html
[46] Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009. p. 248-255. Available from: https://doi.org/10.1109/CVPR.2009.5206848
[47] Breiman L. Random Forests. Machine Learning. 2001;45(1):5-32. Available from: https://doi.org/10.1023/A:1010933404324
[48] Cortes C, Vapnik V. Support-vector networks. Machine Learning. 1995;20(3):273-297. Available from: https://doi.org/10.1007/BF00994018
[49] Jain AK. Data clustering: 50 years beyond K-means. Pattern Recognition Letters. 2010;31(8):651-666. Available from: https://doi.org/10.1016/j.patrec.2009.09.011
[50] Zadeh LA. Fuzzy sets. Information and Control. 1965;8(3):338-353. Available from: https://doi.org/10.1016/S0019-9958(65)90241-X
[51] Kohonen T. The self-organizing map. Proceedings of the IEEE. 2002;78(9):1464-1480. Available from: https://doi.org/10.1109/5.58325
[52] Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. Available from: https://doi.org/10.1007/978-0-387-84858-7
[53] Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. Advances in Neural Information Processing Systems. 2014;27. Available from: https://doi.org/10.1145/3422622
[54] Shorten C, Khoshgoftaar TM. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data. 2019;6(1):1-48. Available from: https://doi.org/10.1186/s40537-019-0197-0
[55] Isensee F, Jaeger PF, Kohl SA, Petersen J, Maier-Hein KH. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. *Nature Methods*. 2021;18(2):203-211. Available from: https://www.nature.com/articles/s41592-020-01008-z
[56] Hatamizadeh A, Nath V, Tang Y, Yang D, Roth HR, Xu D. Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images. International MICCAI Brainlesion Workshop. 2021. p. 272-284. Cham: Springer International Publishing. Available from: https://link.springer.com/chapter/10.1007/978-3-031-08999-2_22
[57] Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, Lu L, Yuille AL, Zhou Y. TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation. arXiv preprint arXiv:2102.04306. 2021. Available from: https://doi.org/10.48550/arXiv.2102.04306
[58] Doshi-Velez F, Kim B. Towards A Rigorous Science of Interpretable Machine Learning. *arXiv preprint arXiv:1702.08608*. 2017. Available from: https://doi.org/10.48550/arXiv.1702.08608
[59] Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the IEEE International Conference on Computer Vision. 2017. p. 618-626. Available from: https://doi.org/10.1109/ICCV.2017.74
[60] Lundberg SM, Lee SI. A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems. 2017;30. Available from: https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html
[61] Akuthota S. Enhancing Chronic Obstructive Pulmonary Disease (COPD) Diagnosis through Machine Learning Models Trained on Respiratory Sounds. *2025 2nd International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE)*. 2025. p. 1-6. Available from: https://doi.org/10.1109/RMKMATE64874.2025.11042677
[62] Akuthota S. Enhanced Breast Cancer Classification Using Attention-Augmented CNN and Multi-View Learning on the Inbreast Dataset. 2025 International Conference on Computing Technologies & Data Communication (ICCTDC). 2025. p. 1-5. Available from: https://doi.org/10.1109/ICCTDC64446.2025.11159034
[63] Ramadevi P, Das R. An Extensive Analysis of Machine Learning Techniques With Hyper-Parameter Tuning by Bayesian Optimized SVM Kernel for the Detection of Human Lung Disease. IEEE Access. 2024;12:97752-97770. Available from: https://doi.org/10.1109/ACCESS.2024.3422449
[64] Manohar B, Das R, Ramadevi P, Balusamy B. A hybridized long-short-term memory networks-based deep learning model using reptile search optimization for COVID-19 prediction. Emerging Trends and Applications of Deep Learning for Biomedical Data Analysis. Academic Press; 2025. p. 49-72. Available from: https://doi.org/10.1016/B978-0-443-26765-9.00003-2
[65] Manohar B, Das R, Lakshmi M. A hybridized LSTM-ANN-RSA based deep learning models for prediction of COVID-19 cases in Eastern European countries. Expert Systems with Applications. 2024;256:124977. Available from: https://doi.org/10.1016/j.eswa.2024.124977
[66] Manohar B, Das R. Comparison of Hybrid Artificial Neural Networks With GA, PSO, and RSA in Predicting COVID-19 Cases: A Case Study of India. Multi-Disciplinary Applications of Fog Computing: Responsiveness in Real-Time. IGI Global Scientific Publishing; 2023. p. 207-244. Available from: https://doi.org/10.4018/978-1-6684-4466-5.ch011
[67] Basholli F, Hayal MR, Elsayed EE, Juraev DA. Deep Learning for Skin Disease Classification: A Comparative Study of CNN and CNN-LSTM Architectures. Journal of Computing and Data Technology. 2025;1(1):40-49. Available from: https://doi.org/10.71426/jcdt.v1.i1.pp40-49
[68] Aazad SK, Saini T, Ajad A, Chaudhary K, Elsayed EE. Deciphering Blood Cells - Method for Blood Cell Analysis using Microscopic Images. Journal of Modern Technology. 2024;1(1):9-18. Available from: https://doi.org/10.71426/jmt.v1.i1.pp9-18
[69] Kalnoor G, Dasari KS, Suma S, Waddenkery N, Pragathi B. Enhanced Brain Tumor Detection from MRI Scans Using Frequency Domain Features and Hybrid Machine Learning Models. Journal of Modern Technology. 2025;1(2):141-149. Available from: https://doi.org/10.71426/jmt.v1.i2.pp141-149
[70] Balaji VR, Jahan MI, Sridarshini T, Geerthana S, Thirumurugan A, Hegde G, Sitharthan R, Dhanabalan SS. Machine learning enabled 2D photonic crystal biosensor for early cancer detection. Measurement. 2024;224:113858. Available from: https://doi.org/10.1016/j.measurement.2023.113858
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Ramadevi Potharla, Bala Krishnama Manohar, N. Bhargavi, Valluru Prathyusha, P. Shailaja (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
