A Novel Deep Learning Approach for Retinopathy Prediction Using Multimodal Data Fusion
Keywords:
Retinopathy, Multimodal, Deep Learning, ClusteringAbstract
In contemporary research on mild cognitive disorders (MCI) and Alzheimer's disease (AD), the predominant approach involves the utilization of double data modalities for making predictions related to AD stages. However, there is a growing recognition of the potential benefits that could be derived from the fusion of multiple data modalities to obtain a more comprehensive perspective in the analysis of AD staging. To address this, we have employed deep learning techniques to holistically assess data from various sources, including, genetic (single nucleotide polymorphisms (SNPs)), imaging (magnetic resonance imaging (MRI)), and clinical tests, with the objective of categorizing patients into distinct groups: AD, MCI, and controls (CN). For the analysis of imaging data, convolutional neural networks have been employed. Moreover, we have introduced a novel approach for data interpretation, enabling the identification of the most influential features learned by these deep models. This interpretation process incorporates clustering and perturbation analysis, shedding light on the crucial aspects of the data contributing to our classification results. Our experimentation, conducted on the dataset (i.e., ADNI), has yielded compelling results. Furthermore, our findings have underscored the significant advantage of integrating multi-modality data over solely relying on double modality models, as it has led to improvements in terms of accuracy, precision, recall, and mean F1 scores.
Downloads
References
Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402–2410 (2016).
Ting, D. S. W. et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 318, 2211–2223 (2017).
Esteva, A. et al. Dermatologist-level classifcation of skin cancer with deep neural networks. Nature 542, 115 (2017).
Weng, S., Xu, X., Li, J. & Wong, S. T. Combining deep learning and coherent anti-Stokes Raman scattering imaging for automated diferential diagnosis of lung cancer. J. Biomed. Opt. 22, 106017 (2017).
Suk, H.-I. & Shen, D. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2013 583–590 (Springer, New York, 2013).
Liu, S. et al. Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer’s disease. Biomed. Eng. IEEE Trans. 62, 1132–1140 (2015).
Suk, H. I., Lee, S. W., Shen, D. & Alzheimer’s Disease Neuroimaging Initiative. Deep sparse multi-task learning for feature selection in Alzheimer’s disease diagnosis. Brain Struct. Funct. 221(5), 2569–2587 (2016).
Schulam, P., Wigley, F. & Saria, S. In AAAI, 2956–2964 (2015).
Suk, H.-I. & Shen, D. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 583–590 (Springer, New York, 2013).
Choi, E., Bahadori, M.T. & Sun, J. Doctor ai: Predicting clinical events via recurrent neural networks. arXiv preprint arXiv:1511.05942 (2015).
Zhou, J. & Troyanskaya, O. G. Predicting efects of noncoding variants with deep learning-based sequence model. Nat. Methods 12, 931–934 (2015).
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H. & Ng, A. Y. Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) 689–696 (2011).
Alzheimer’s Association. 2016 Alzheimer’s disease facts and fgures. Alzheimer’s Dement. 12(4), 459–509 (2016).
Alzheimer’s Association. 2013 Alzheimer’s disease facts and fgures. Alzheimer’s Dement. 9(2), 208–245 (2013).
Patterson, C. World Alzheimer Report 2018—Te State of the Art of Dementia Research: New Frontiers. (Alzheimer’s Disease International (ADI), London, 2018).
Perrin, R. J., Fagan, A. M. & Holtzman, D. M. Multimodal techniques for diagnosis and prognosis of Alzheimer’s disease. Nature 461, 916–922 (2009).
Blennow, K. et al. Clinical utility of cerebrospinal fuid biomarkers in the diagnosis of early Alzheimer’s disease. Alzheimer’s Dement. 11, 58–69 (2015).
Eskildsen, S. F. et al. Structural imaging biomarkers of Alzheimer’s disease: Predicting disease progression. Neurobiol. Aging 36, S23–S31 (2015).
Grimmer, T. et al. Visual versus fully automated analyses of 18F-FDG and amyloid PET for prediction of dementia due to Alzheimer disease in mild cognitive impairment. J. Nucl. Med. 57, 204–207 (2016).
Cui, R., Liu, M. & Initiative, A. S. D. N. RNN-based longitudinal analysis for diagnosis of Alzheimer’s disease. Comput. Med. Imaging Graph. 73, 1–10 (2019).
Barnes, J. et al. Vascular and Alzheimer’s disease markers independently predict brain atrophy rate in Alzheimer’s Disease Neuroimaging Initiative controls. Neurobiol. Aging 34, 1996–2002 (2013).
Doecke, J. D. et al. BLood-based protein biomarkers for diagnosis of alzheimer disease. Arch. Neurol. 69, 1318–1325 (2012).
Lee, G., Nho, K., Kang, B., Sohn, K.-A. & Kim, D. Predicting Alzheimer’s disease progression using multi-modal deep learning approach. Sci. Rep. 9, 1952 (2019).
Zhao, J. et al. Learning from longitudinal data in electronic health record and genetic data to improve cardiovascular event prediction. Sci. Rep. 9, 717 (2019).
Wu, W., Venugopalan, J. & Wang, M. D. 11C-PIB PET image analysis for Alzheimer’s diagnosis using weighted voting ensembles.
In 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 3914–3917 (IEEE, 2017).
Phan, J. H., Quo, C. F. & Wang, M. D. Functional genomics and proteomics in the clinical neurosciences: data mining and bioinformatics. Prog. Brain Res. 158, 83–108 (2006).
Dyrba, M., Grothe, M., Kirste, T. & Teipel, S. J. Multimodal analysis of functional and structural disconnection in Alzheimer’s disease using multiple kernel SVM. Hum. Brain Mapp. 36, 2118–2131 (2015).
Shafer, J. L. et al. Predicting cognitive decline in subjects at risk for Alzheimer disease by using combined cerebrospinal fuid, MR imaging, and PET biomarkers. Radiology 266, 583–591 (2013).
Dai, Z. et al. Discriminative analysis of early Alzheimer’s disease using multi-modal imaging and multi-level characterization with multi-classifer (M3). NeuroImage 59, 2187–2195 (2012).
Dyrba, M. et al. Predicting prodromal Alzheimer’s disease in subjects with mild cognitive impairment using machine learning classifcation of multimodal multicenter difusion-tensor and magnetic resonance imaging data. J. Neuroimaging 25, 738–747 (2015).
Lorenzi, M. et al. Multimodal image analysis in Alzheimer’s disease via statistical modelling of non-local intensity correlations. Sci. Rep. 6, 22161 (2016).
Vogel, J. W. et al. Brain properties predict proximity to symptom onset in sporadic Alzheimer’s disease. Brain 141, 1871–1883 (2018).
Gray, K. R., Aljabar, P., Heckemann, R. A., Hammers, A. & Rueckert, D. Random forest-based similarity measures for multi-modal classifcation of Alzheimer’s disease. NeuroImage 65, 167–175 (2013).
Zhang, D., Wang, Y., Zhou, L., Yuan, H. & Shen, D. Multimodal classifcation of Alzheimer’s disease and mild cognitive impairment. NeuroImage 55, 856–867 (2011).
Wang, H. et al. Identifying disease sensitive and quantitative trait-relevant biomarkers from multidimensional heterogeneous imaging genetics data via sparse multimodal multitask learning. Bioinformatics 28, i127–i136 (2012).
Suk, H.-I., Lee, S.-W. & Shen, D. Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. NeuroImage 101, 569–582 (2014).
Mueller, S. G. et al. Ways toward an early diagnosis in Alzheimer’s disease: the Alzheimer’s Disease Neuroimaging Initiative (ADNI). Alzheimer’s Dement. 1, 55–66 (2005).
Lahat, D., Adali, T., & Jutten, C. (2015). Multimodal data fusion: an overview of methods, challenges, and prospects. Proceedings of the IEEE, 103(9), 1449-1477.
Hsu, M. Y., Chiou, J. Y., Liu, J. T., Lee, C. M., Lee, Y. W., Chou, C. C., ... & Tseng, V. S. (2021). Deep learning for automated diabetic retinopathy screening fused with heterogeneous data from EHRs can lead to earlier referral decisions. Translational Vision Science & Technology, 10(9), 18-18.
Sanborn, G. E., & Wroblewski, J. J. (2018). Evaluation of a combination digital retinal camera with spectral-domain optical coherence tomography (SD-OCT) that might be used for the screening of diabetic retinopathy with telemedicine: a pilot study. Journal of Diabetes and its Complications, 32(11), 1046-1050.
Aiello, L. P., Cahill, M. T., & Wong, J. S. (2001). Systemic considerations in the management of diabetic retinopathy. American journal of ophthalmology, 132(5), 760-776.
Buzzetti, E., Pinzani, M., & Tsochatzis, E. A. (2016). The multiple-hit pathogenesis of non-alcoholic fatty liver disease (NAFLD). Metabolism, 65(8), 1038-1048.
Hageman, G. S., Luthert, P. J., Chong, N. V., Johnson, L. V., Anderson, D. H., & Mullins, R. F. (2001). An integrated hypothesis that considers drusen as biomarkers of immune-mediated processes at the RPE-Bruch's membrane interface in aging and age-related macular degeneration. Progress in retinal and eye research, 20(6), 705-732.
Onnela, J. P., & Rauch, S. L. (2016). Harnessing smartphone-based digital phenotyping to enhance behavioral and mental health. Neuropsychopharmacology, 41(7), 1691-1696.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
