3D Visualization for Lung Surface Images of Covid-19 Patients based on U-Net CNN Segmentation

  • FX Ferdinandus Department of Electrical Engineering, Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia
  • Esther Irawati Setiawan Department of Informatics, Institut Sains dan Teknologi Terpadu Surabaya, Surabaya, Indonesia
  • Eko Mulyanto Yuniarno Department of Computer Engineering, Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia
  • Mauridhi Hery Purnomo University Center of Excellence on Artificial Intelligence for Healthcare and Society (UCE AIHeS) Surabaya, Indonesia
Keywords: Covid-19, CT-scan, Deep Learning, Lung segmentation, U-Net

Abstract

The Covid-19 infection challenges medical staff to make rapid diagnoses of patients. In just a few days, the Covid-19 virus infection could affect the performance of the lungs. On the other hand, semantic segmentation using the Convolutional Neural Network (CNN) on Lung CT-scan images had attracted the attention of researchers for several years, even before the Covid-19 pandemic. Ground Glass Opacity (GGO), in the form of white patches caused by Covid-19 infection, is detected inside the patient’s lung area and occasionally at the edge of the lung, but no research has specifically paid attention to the edges of the lungs. This study proposes to display a 3D visualization of the lung surface of Covid-19 patients based on CT-scan image segmentation using U-Net architecture with a training dataset from typical lung images. Then the resulting CNN model is used to segment the lungs of Covid-19 patients. The segmentation results are selected as some slices to be reconstructed into a 3D lung shape and displayed in 3D animation. Visualizing the results of this segmentation can help medical staff diagnose the lungs of Covid-19 patients, especially on the surface of the lungs of patients with GGO at the edges. From the lung segmentation experiment results on ten patients in the Zenodo dataset, we have a Mean-IoU score = of 76.86%, while the visualization results show that 7 out of 10 patients (70%) have eroded lung surfaces. It can be seen clearly through 3D visualization.

Downloads

Download data is not yet available.

References

Nifty format, [Online]. Available: http://nifty.nimh.nih.gov.

W. Zhao, W. Jiang, and X. Qiu, Deep learning for COVID-19 detection based on CT images, Sci Rep, vol. 11, no. 1, Dec. 2021, doi: 10.1038/s41598-021-93832-2. DOI: https://doi.org/10.1038/s41598-021-93832-2

Jeremy Jordan, An Overview of Semantic Image Segmentation, https://www.jeremyjordan.me/semantic-segmentation, May 21, 2018.

X. Liu, Z. Deng, and Y. Yang, Recent progress in semantic image segmentation, Artif Intell Rev, vol. 52, no. 2, pp. 1089–1106, Aug. 2019, doi: 10.1007/s10462-018-9641-3. DOI: https://doi.org/10.1007/s10462-018-9641-3

I. Ulku and E. Akagunduz, A Survey on Deep Learning-based Architectures for Semantic Segmentation on 2D images, Dec. 2019, doi: 10.1080/08839514.2022.2032924. DOI: https://doi.org/10.1080/08839514.2022.2032924

O. Ronneberger, P. Fischer, and T. Brox, U-net: Convolutional networks for biomedical image segmentation, in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015, vol. 9351, pp. 234–241. doi: 10.1007/978-3-319-24574-4_28. DOI: https://doi.org/10.1007/978-3-319-24574-4_28

F. Ferdinandus, E. Mulyanto Yuniarno, I. Ketut Eddy Purnama, and M. Hery Purnomo, Covid-19 Lung Segmentation using U-Net CNN based on Computed Tomography Image, in 2022 IEEE 9th International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), June 2022. DOI: https://doi.org/10.1109/CIVEMSA53371.2022.9853695

J. F. W. Chan et al., A familial cluster of pneumonia associated with the 2019 novel coronavirus indicating person-to-person transmission: a study of a family cluster, The Lancet, vol. 395, no. 10223, pp. 514–523, Feb. 2020, doi: 10.1016/S0140-6736(20)30154-9. DOI: https://doi.org/10.1016/S0140-6736(20)30154-9

F. Shi et al., Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation, and Diagnosis for COVID-19, IEEE Reviews in Biomedical Engineering, vol. 14. Institute of Electrical and Electronics Engineers Inc., pp. 4–15, 2021. doi: 10.1109/RBME.2020.2987975. DOI: https://doi.org/10.1109/RBME.2020.2987975

D. P. Fan et al., Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images, IEEE Trans Med Imaging, vol. 39, no. 8, pp. 2626–2637, Aug. 2020, doi: 10.1109/TMI.2020.2996645. DOI: https://doi.org/10.1109/TMI.2020.2996645

T. Karlita, E. M. Yuniarno, I. K. E. Purnama, and M. H. Purnomo, Detection of COVID-19 on Chest X-Ray Images using Inverted Residuals Structure-Based Convolutional Neural Networks, in 2020 3rd International Conference on Information and Communications Technology, ICOIACT 2020, Nov. 2020, pp. 371–376. doi: 10.1109/ICOIACT50329.2020.9332153. DOI: https://doi.org/10.1109/ICOIACT50329.2020.9332153

A. Kalinovsky and V. Kovalev, Lung Image Segmentation Using Deep Learning Methods and Convolutional Neural Networks, 2016. [Online]. Available: http://imlab.grid.by/

S. G. C, V. S. T, R. v Gowda, P. R. Udupa, and S. Reddy, A Machine learning Classification approach for detection of Covid 19 using CT images, EMITTER International Journal of Engineering Technology, vol. 10, no. 1, pp. 183–194, 2022, doi: 10.24003/emitter.v10i1.672. DOI: https://doi.org/10.24003/emitter.v10i1.672

B. Ait Skourt, A. el Hassani, and A. Majda, Lung CT image segmentation using deep neural networks, in Procedia Computer Science, 2018, vol. 127, pp. 109–113. doi: 10.1016/j.procs.2018.01.104. DOI: https://doi.org/10.1016/j.procs.2018.01.104

T. Tolxdorff, T. M. Deserno Heinz Handels, and A. H. Maier Klaus Maier-Hein Christoph Palm Hrsg, Bildverarbeitung für die Medizin 2020, 2020. [Online]. Available: http://www.springer.com/series/2872 DOI: https://doi.org/10.1007/978-3-658-29267-6

Adhiparasakthi Engineering College. Department of Electronics and Communication Engineering, Institute of Electrical and Electronics Engineers. Madras Section, and Institute of Electrical and Electronics Engineers, Proceedings of the 2018 IEEE International Conference on Communication and Signal Processing (ICCSP) : 3rd - 5th April 2018, Melmaruvathur, India. 2018.

H. Shaziya and K. Shyamala, Pulmonary CT Images Segmentation using CNN and UNet Models of Deep Learning, in 2020 IEEE Pune Section International Conference, PuneCon 2020, Dec. 2020, pp. 195–201. doi: 10.1109/PuneCon50868.2020.9362463. DOI: https://doi.org/10.1109/PuneCon50868.2020.9362463

K. K. Bressem, S. M. Niehues, B. Hamm, M. R. Makowski, J. L. Vahldiek, and L. C. Adams, 3D U-Net for segmentation of COVID-19 associated pulmonary infiltrates using transfer learning: State-of-the-art results on affordable hardware, Jan. 2021, [Online]. Available: http://arxiv.org/abs/2101.09976 DOI: https://doi.org/10.21203/rs.3.rs-259319/v1

E. Martínez Chamorro, A. Díez Tascón, L. Ibáñez Sanz, S. Ossaba Vélez, and S. Borruel Nacenta, Radiologic diagnosis of patients with COVID-19, Radiología (English Edition), vol. 63, no. 1, pp. 56–73, Jan. 2021, doi: 10.1016/j.rxeng.2020.11.001. DOI: https://doi.org/10.1016/j.rxeng.2020.11.001

D. Cozzi et al., Ground-glass opacity (GGO): a review of the differential diagnosis in the era of COVID-19, Japanese Journal of Radiology, vol. 39, no. 8. Springer Japan, pp. 721–732, Aug. 01, 2021. doi: 10.1007/s11604-021-01120-w. DOI: https://doi.org/10.1007/s11604-021-01120-w

E. P. I. K. A. D. N. D. Athanasios Voulodimos, Deep learning models for COVID-19 infected area segmentation in CT images, vol. 6, 2021, doi: 10.1101/2020.05.08.20094664. DOI: https://doi.org/10.1101/2020.05.08.20094664

T. Zhou, S. Canu, and S. Ruan, Automatic COVID-19 CT segmentation using U-Net integrated spatial and channel attention mechanism, Int J Imaging Syst Technol, vol. 31, no. 1, pp. 16–27, Mar. 2021, doi: 10.1002/ima.22527. DOI: https://doi.org/10.1002/ima.22527

K Scott Mader, Kaggle Lung Dataset, 2017.

Ma Jun, et all, Zenodo Lung Covid-19 Dataset, https://zenodo.org/record/3757476, Apr. 20, 2020.

Y. H. Nai et al., Comparison of metrics for the evaluation of medical segmentations using prostate MRI dataset, Comput Biol Med, vol. 134, Jul. 2021, doi: 10.1016/j.compbiomed.2021.104497. DOI: https://doi.org/10.1016/j.compbiomed.2021.104497

Published
2022-12-28
How to Cite
FX Ferdinandus, Esther Irawati Setiawan, Eko Mulyanto Yuniarno, & Hery Purnomo, M. (2022). 3D Visualization for Lung Surface Images of Covid-19 Patients based on U-Net CNN Segmentation. EMITTER International Journal of Engineering Technology, 10(2), 320-337. https://doi.org/10.24003/emitter.v10i2.709
Section
Articles