A Review on Methodologies and Challenges of Whole Heart Segmentation Using Deep Learning
Keywords:
Whole heart segmentation, Deep learning, neural networks, cardiac image analysis, MRI, CTAbstract
In recent years, deep-learning approaches have had an enormous impact on the analysis of medical images, particularly in whole-heart segmentation. This active research focuses on the accurate delineation of substructures within the heart, the accurate assessment of cardiac function, treatment planning, and facilitating cardiac interventional procedures. It also provides essential morphological information. Ultrasound, Magnetic Resonance Imaging (MRI), and Computed Tomography (CT) are the widely used imaging modalities to segment the substructures such as the ventricles, Atria, and Aorta within the heart. The conduct of whole-heart segmentation is challenging because of laborious manual delineation, which is tedious, subject to change, and requires meticulous analysis. This article discusses the recent advancements in whole-heart segmentation and provides a comprehensive analysis of the various deep learning approaches employed in this domain. This paper highlights different deep learning models such as U-Net, V-Net, and attention mechanisms used to achieve accurate segmentation of the whole heart. Additionally, this paper also explores the volume rendering methods that can be applied to heart structures and the advantages and limitations of these methods in obtaining accuracy and robustness in handling complex cardiac substructures. This article identifies these challenges and suggests research directions to promote reproducible and robust results.
Downloads
References
Geneva: World Health Organization; 2016 (WHO/DAD/2016.1). World health statistics overview 2016: Monitoring health for the SDGs (sustainable development goals) https://www.who.int/gho/ publications/world_health_statistics/2016/en/. Accessed: September 16, 2019.
A. Chartsias, T. Joyce, G. Papanastasiou, S. Semple, M. Williams, D. Newby, R. Dharmakumar, and S. A. Tsaftaris, Factorized spatial representation learning: Application in semi-supervised myocardial segmentation,’’ in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Intervent. Cham, Switzerland: Springer, 2018, pp. 490–498.
C. Xu, L. Xu, G. Brahm, H. Zhang, and S. Li, ‘‘MuTGAN: Simultaneous segmentation and quantification of myocardial infarction without contrast agents via joint adversarial learning,’’ in Proc. Int. Conf. Med. Image Comput. Comput. Assist. Intervent. Cham, Switzerland: Springer, 2018, pp. 525–534.
H.-Y. Lee, N. C. F. Codella, M. D. Cham, J. W. Weinsaft, and Y. Wang, ‘‘Automatic left ventricle segmentation using iterative thresholding and an active contour model with adaptation on short-axis cardiac MRI,’’ IEEE Trans. Biomed. Eng., vol. 57, no. 4, pp. 905–913, Apr. 2010.
L. Zhang, M. Pereaez, S. K. Piechnik, S. Neubauer, S. E. Petersen, and A. F. Frangi, ‘‘Multi-input and dataset-invariant adversarial learning (MDAL) for left and right-ventricular coverage estimation in cardiac MRI,’’ in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Intervent. Cham, Switzerland: Springer, Sep. 2018, pp. 481–489.
Y. Mo, F. Liu, D. McIlwraith, G. Yang, J. Zhang, T. He, and Y. Guo, ‘‘The Deep Poincaré Map: A Novel Approach for Left Ventricle Segmentation,’’ in Proc. Int. Conf. Med. Image Comput. Comput.-Assist.Intervent. Cham, Switzerland: Springer, 2018, pp. 561–568.
O. Ronneberger, P. Fischer, and T. Brox, ‘‘U-net: Convolutional networks for biomedical image segmentation,’’ in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Intervent. Cham, Switzerland: Springer, 2015, pp. 234–241.
W. Xiancheng, L. Wei, M. Bingyi, J. He, Z. Jiang, W. Xu, Z. Ji, G. Hong, and S. Zhaomeng, ‘‘Retina blood vessel segmentation using a U-net- based Convolutional Neural Network,’’inProc.Procedia Comput. Sci., Int. Conf. Data Sci., Beijing, China, Jun. 2018, pp. 8–9.
C.Playout, R.Duval, and F.Cheriet,‘‘A multitask learning architecture for simultaneous segmentation of bright and red lesions in fundus images,’’ in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Intervent. Cham, Switzerland: Springer, 2018, pp. 101–108.
Y. Zhang and A. C. S. Chung, ‘‘Deep supervision with additional labels for retinal vessel segmentation task,’’ in Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Intervent. Cham, Switzerland: Springer, 2018, pp. 83–91.
H. Dong, G. Yang, F. Liu, Y. Mo, and Y. Guo, ‘‘Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks,’’ in Proc. Annu. Conf. Med. Image Understand. Anal. Cham, Switzerland: Springer, 2017, pp. 506–517.
Wang, C., Smedby, O. 2017. Automatic whole heart segmentation using deep learning and shape context. In: International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 242–249.
Hortense A. Kirisli, Michiel Schaap, Stefan Klein, Lisan A. Neefjes, Annick C. Weustink, Theo Van Walsum, and Wiro J. Niessen "Fully automatic cardiac segmentation from 3D CTA data: a multi-atlas based approach", Proc. SPIE 7623, Medical Imaging 2010: Image Processing, 762305 (12 March 2010); https://doi.org/10.1117/12.838370
Yang, Guanyu et al. “Automatic Whole Heart Segmentation in CT Images Based on Multi-atlas Image Registration.” STACOM@MICCAI (2017).
Zhuang X., Bai W., Song J., Zhan S., Qian X., Shi W., Lian Y., Rueckert D. Multiatlas whole heart segmentation of CT data using conditional entropy for atlas ranking and selection Med. Phys., 42 (7) (2015), pp. 3822-3833
M.P.Heinrich and J. Oster, MRII whole heart segmentation using discrete nonlinear registration and fast non-local fusion,’’ in Proc. Int. Workshop Stat. Atlases Comput. Models Heart. Cham, Switzerland: Springer, 2017, pp. 233–241.
X. Zhuang and J. Shen, ‘‘Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI,’’ Med. Image Anal., vol. 31, pp. 77–87, Jul. 2016.
G. Galisot, T. Brouard, and J.-Y. Ramel, ‘‘Local probabilistic atlases and a posteriori correction for the segmentation of heart images,’’ in Proc. Int. Workshop Stat. Atlases Comput. Models Heart. Cham, Switzerland: Springer, 2017, pp. 207–214.
Kang, H.C., Kim, B., Lee, J. et al. Automatic left and right heart segmentation using power watershed and active contour model without edge. Biomed. Eng. Lett. 4, 355–361 (2014). https://doi.org/10.1007/s13534-014-0164-9.
Ho Chul Kang, Jeongjin Lee, Juneseuk Shin, Automatic Four-Chamber Segmentation Using Level-Set Method and Split Energy Function, Healthc Inform Res. 2016;22(4):285-292.
X. Zhao, Y. Wang and G. Jozsef, "Robust shape-constrained active contour for whole heart segmentation in 3-D CT images for radiotherapy planning," 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, 2014, pp. 1-5, doi: 10.1109/ICIP.2014.7024999.
Lee HY, Codella NC, Cham MD, Weinsaft JW, Wang Y. Automatic left ventricle segmentation using iterative thresholding and an active contour model with adaptation on short-axis cardiac MRI. IEEE Trans Biomed Eng. 2010 Apr;57(4):905-13. doi: 10.1109/TBME.2009.2014545. Epub 2009 Feb 6. PMID: 19203875.
Katherine et al 2021 J. Phys.: Conf. Ser. 1816 012080. DOI 10.1088/1742-6596/1816/1/012080
Rodríguez R. A strategy for blood vessels segmentation based on the threshold which combines statistical and scale space filter. Application to the study of angiogenesis. Comput Methods Programs Biomed. 2006 Apr;82(1):1-9. doi: 10.1016/j.cmpb.2005.10.008. Epub 2006 Mar 23. PMID: 16556472.
Marija Habijan, Danilo Babin, Irena Gali´c, Hrvoje Leventi´c, Kreˇsimir Romi´c, Lazar Velicki, Aleksandra Piˇzurica, Overview of the whole heart and heart chamber segmentation methods, Cardiovasc. Eng. Technol. (2020) 1–23.
Mortazi,A., Burt, J., Bagci, U., 2017. Multi-planar deep segmentation networks for cardiac substructures from MRI and CT. In: International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 199–206.
Yang, X., Bian, C., Yu, L., Ni, D.,Heng, P.-A., 2017. 3D convolutional networks for fully automatic fine-grained whole heart partition. In: International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 181–189.
Tong, Q. , Ning, M. , Si, W. , Liao, X. , Qin, J. , 2017. 3D deeply-supervised U-Net based whole heart segmentation. In: International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 224–232.
Yang, G. , Sun, C. , Chen, Y. , Tang, L. , Shu, H. , Dillenseger, J.-l. , 2017. Automatic whole heart segmentation in CT images based on multi-atlas mage registration. In: International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 250–257.
Wang, C., Smedby, O. 2017. Automatic whole heart segmentation using deep learning and shape context. In: International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 242–249.
Shi, Z. et al. (2018). Bayesian VoxDRN: A Probabilistic Deep Voxelwise Dilated Residual Network for Whole Heart Segmentation from 3D MR Images. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science(), vol 11073. Springer, Cham. https://doi.org/10.1007/978-3-030-00937-3_65
Xu, Zhanwei, Ziyi Wu, and Jianjiang Feng. "CFUN: Combining faster R-CNN and U-net network for efficient whole heart segmentation." arXiv preprint arXiv:1812.04914 (2018).
Yang, X. , Bian, C. , Yu, L. , Ni, D. , Heng, P.-A. , 2017. Hybrid loss guided convolutional networks for whole heart parsing. In: International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 215–223.
M. Habijan, H. Leventić, I. Galić and D. Babin, "Whole Heart Segmentation from CT images Using 3D U-Net architecture," 2019 International Conference on Systems, Signals and Image Processing (IWSSIP), Osijek, Croatia, 2019, pp. 121-126, doi: 10.1109/IWSSIP.2019.8787253.
Mahendra Khened , Alex Varghese, Ganapathy Krishnamurthi: Fully convolutional multi-scale residual DenseNets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers. Medical Image Anal. 51: 21-45 (2019)
Christian Payer, Darko Stern, Horst Bischof, and Martin Urschler “Multi-Label Whole Heart Segmentation Using CNNs and Anatomical Label Configurations”, Statistical Atlases and Computational Models of the Heart. ACDC and MMWHS Challenges, Springer 2019.
C. Ye, W. Wang, S. Zhang and K. Wang, "Multi-Depth Fusion Network for Whole-Heart CT Image Segmentation," in IEEE Access, vol. 7, pp. 23421-23429, 2019, doi: 10.1109/ACCESS.2019.2899635.
Tao Liu, Yun Tian, Shifeng Zhao, And Qingjun Wang “Automatic Whole Heart Segmentation Using a Two-Stage U-Net Framework and an Adaptive Threshold Window”, IEEE VOLUME 7, 2020.
Kening Le, Zeyu Lou, Weiliang Huo and Xiaolin Tian, “Auto Whole Heart Segmentation from CT images Using an Improved Unet-GAN”, Journal of Physics: Conference Series 1769 (2021) 012016,doi:10.1088/1742-6596/1769/1/012016
Sulaiman Vesal, Andreas Maier, “Fully Automated 3D Cardiac MRI Localisation and Segmentation Using Deep Neural Networks”, Journal of Imaging, 6(7), 65, 2020.
Z. Lou, W. Huo, K. Le and X. Tian, "Whole Heart Auto Segmentation of Cardiac CT Images Using U-Net Based GAN," 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Chengdu, China, 2020, pp. 192-196, doi: 10.1109/CISP-BMEI51763.2020.9263532.
Galea, R.; Diosan, L.; Andreica, A.; Popa, L.; Manole, S.; Bálint, Z. Region-of-Interest Based Cardiac Image Segmentation with Deep Learning. Appl. Sci. 2021, 11, 1965. https://doi.org/10.3390/ app11041965
Habijan, M.; Galic´, I.; Leventic´, L.; Romic´, K. Whole Heart Segmentation Using 3D FM-Pre-ResNet Encoder–Decoder Based Architecture with Variational Autoencoder Regularization. Appl. Sci. 2021, 11, 3912. https://doi.org/ 10.3390/app11093912
Xu, X. et al. (2020). ImageCHD: A 3D Computed Tomography Image Dataset for Classification of Congenital Heart Disease. In: , et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12264. Springer, Cham. https://doi.org/10.1007/978-3-030-59719-1_8
Yuhui Songb, Xiuquan Dua,Yanping Zhanga, Shuo Li Two-stage segmentation network with feature aggregation and multi-level attention mechanism for multi-modality heart images, Elsevier (2022)
Wang, K., Zhang, L., Gai, C., and Zuo, W., Illustrative visualization of segmented human cardiac anatomy based on context-preserving model. In: Computing in Cardiology, pp. 485– 488, 2011.
Yang, F., Zuo, W., Wang, K., and Zhang, H., Visualization of segmented cardiac anatomy with accelerated rendering method. In: Computing in Cardiology, IEEE, pp. 789–792, 2009.
Kurzendorfer, T., Girard, E., Gralewski, K., Kleinoeder, A., Kiraly, A. P., Strobel, N., and Dori, Y., New biplane x-ray magnetic resonance image fusion prototype for 3d enhanced cardiac catheterization in congenital heart diseases. J. Cardiovasc. Magn. Reson. 16(Suppl1):O103, 2014
Gai, C., Wang, K., Zhang, L., and Zuo, W., Strategy of statistics based visualization for segmented 3d cardiac volume dataset. Adv. Int. Comp. 6838:250–256, 2012.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
