Satyadhyan CHICKERUR, Mahesh S PATIL, Yeshwanth Kumar VS, Vijayalakshmi A BAKALE, Shantala GIRADDI, Vivekanand C ROODAGI, Yashaswini N KULKARNI

Deep hyperparameter transfer learning for diabetic retinopathy classification

The detection of diabetic retinopathy (DR) in millions of diabetic patients across the globe is a challenging problem. Diagnosis of retinopathy is a lengthy and tedious process, requiring a medical professional to assess the individual fundus images of a patient’s retina. This process can be automated by applying deep learning (DL) technology given a huge dataset. The problems associated with DL are the unavailability of a large dataset and their higher training time. The DL model’s best performance is achieved using set of optimal hyperparameters (OHPs) obtained by performing costly iterations of hyperparameter optimization (HPO). These problems can be addressed by using transfer learning (TL) technique in both DL model training and HPO. TL in HP tuning is the focus of this work. The authors study the applicability of EyePACS DR dataset’s OHPs to other DR datasets, forming the basis of the research question addressed in this work. The DR classification is performed using a ResNet model trained on the EyePACS (kaggle) and Indian diabetic retinopathy image dataset (IDRiD) datasets. Various HPs tuned in this work are data augmentation configuration, number of layers, optimizers, data samplers, learning rate, and momentum. The authors demonstrate that EyePACS dataset’s OHPs are suitable for training with IDRiD dataset without needing to tune HPs for IDRiD dataset from scratch. The OHPs for a task and their reusability is poorly reported in the literature. Therefore, the EyePACS DR dataset’s OHPs reported here can be used by other researchers. Moreover, the researchers working on other DR datasets can also apply the same OHPs since they are reusable and no iterations of HPO are required. The OHPs are provided for both EyePAC and IDRiD datasets after being tuned from scratch, which can be used as starting point for HPO by others.

PDF

___

[1] Kaiming H, Zhang X,Ren S, Sun J. Identity mappings in deep residual networks. In: European Conference on Computer Vision. Springer, Cham; 2016. pp. 630-645.
[2] Xiaogang L, Tiantian P, Biao X, Weixiang L, Liang P et al. Convolutional neural networks based transfer learning for diabetic retinopathy fundus image classification. In: IEEE 2017 10th International Congress on Image and Signal Processing, Biomedical Engineering and Informatics (CISP-BMEI); 2017. pp. 1-11.
[3] Sharif Razavian A, Hossein A, Josephine S, Stefan C. CNN features off-the-shelf: an astounding baseline for recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops; 2014. pp. 806-813.
[4] Oquab M, Leon B, Ivan L,Josef S. Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014. pp. 1717-1724.
[5] Tamaki T, Junki Y, Misato K, Bisser R, Kaneda K et al. Computer-aided colorectal tumor classification in NBI endoscopy using local features. Medical Image Analysis 2013; 17(1): 78-100.
[6] Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N et al. Decaf: A deep convolutional activation feature for generic visual recognition. In: International Conference on Machine Learning; PMLR; 2014. pp. 647-655.
[7] Bergstra J, Bardenet R, Bengio Y, Kégl B. Algorithms for hyperparameter optimization. Advances in Neural Information Processing Systems 2011; 24.
[8] Bergstra J, Bengio Y. Random search for hyper-parameter optimization. Journal of Machine Learning Research 2012; 13 (2).
[9] Shankar K, Zhang Y, Liu Y, Wu L, Chen CH. Hyperparameter tuning deep learning for diabetic retinopathy fundus image classification. IEEE Access 2020; 8: 118164-118173.
[10] Mohammadian S, Karsaz A, Roshan YM. Comparative study of fine-tuning of pre-trained convolutional neural networks for diabetic retinopathy screening. In: IEEE 2017 24th National and 2nd International Iranian Conference on Biomedical Engineering (ICBME); 2017. pp. 1-6.
[11] Van Rijn JN, Hutter F. Hyperparameter importance across datasets. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; 2018. pp. 2367-2376.
[12] Swersky K, Snoek J, Adams RP. Multi-task Bayesian optimization. Advances in Neural Information Processing Systems 2013; 26: 2004-2012.
[13] Porwal P, Pachade S, Kamble R, Kokare M, Deshmukh G et al. Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathy screening research. Data 2018; 3 (3): 25.
[14] Verma K, Deep P, Ramakrishnan AG. Detection and classification of diabetic retinopathy using retinal images. In: 2011 Annual IEEE India Conference; 2011. pp. 1-6).
[15] Akram MU, Khalid S, Khan SA. Identification and classification of microaneurysms for early detection of diabetic retinopathy. Pattern Recognition 2013; 46 (1): 107-16.
[16] Gargeya R, Leng T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology. 2017; 124 (7): 962-969.
[17] Lam C, Yi D, Guo M, Lindsey T. Automated detection of diabetic retinopathy using deep learning. AMIA Summits on Translational Science Proceedings 2018; 2018: 147.
[18] Bidari I, Chickerur S, Ranmale H, Talawar S, Ramadurg H et al. Hyperspectral imagery classification using deep learning. In: IEEE 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4); 2020. pp. 672-676.
[19] Khaled O, El-Sahhar M, El-Dine MA, Talaat Y, Hassan YM et al. Cascaded architecture for classifying the preliminary stages of diabetic retinopathy. In: Proceedings of the 2020 9th International Conference on Software and Information Engineering (ICSIE); 2020. pp. 108-112.
[20] Khan RU, Zhang X, Kumar R, Tariq HA. Analysis of resnet model for malicious code detection. In: IEEE 2017 14th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP); 2017. pp. 239-242.
[21] Gulshan V, Peng L, Coram M, Stumpe MC, Wu D et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 2016; 316 (22): 2402-2410.
[22] Zhang D, Bu W, Wu X. Diabetic retinopathy classification using deeply supervised ResNet. In: 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI); 2017. pp. 1-6.
[23] Gardner GG, Keating D, Williamson TH, Elliott AT. Automatic detection of diabetic retinopathy using an artificial neural network: a screening tool. British Journal of Ophthalmology 1996; 80 (11): 940-944.
[24] Weiss K, Khoshgoftaar TM, Wang D. A survey of transfer learning. Journal of Big Data 2016; 3 (1): 1-40.
[25] Zhuang F, Qi Z, Duan K, Xi D, Zhu Y et al. A comprehensive survey on transfer learning. Proceedings of the IEEE. 2020; 109 (1): 43-76.
[26] Islam SM, Hasan MM, Abdullah S. Deep learning based early detection and grading of diabetic retinopathy using retinal fundus images. arXiv preprint arXiv:1812.10595. 2018.
[27] Shankar K, Sait AR, Gupta D, Lakshmanaprabu SK, Khanna A et al. Automated detection and classification of fundus diabetic retinopathy images using synergic deep learning model. Pattern Recognition Letters 2020; 133: 210-216.
[28] Qummar S, Khan FG, Shah S, Khan A, Shamshirband S et al. A deep learning ensemble approach for diabetic retinopathy detection. IEEE Access 2019; 7: 150530-150539.
[29] Hacisoftaoglu RE, Karakaya M, Sallam AB. Deep learning frameworks for diabetic retinopathy detection with smartphone-based retinal imaging systems. Pattern Recognition Letters 2020; 135: 409-417.
[30] Xia Y, Liu C, Li Y, Liu N. A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoring. Expert Systems with Applications 2017; 78: 225-241.
[31] Gonzalez-Cuautle D, Corral-Salinas UY, Sanchez-Perez G, Perez-Meana H, Toscano-Medina K et al. An efficient botnet detection methodology using hyper-parameter optimization trough grid-search techniques. In: IEEE 2019 7th International Workshop on Biometrics and Forensics (IWBF); 2019. pp. 1-6.
[32] Dewancker I, McCourt M, Clark S. Bayesian optimization for machine learning: A practical guidebook. arXiv preprint arXiv:1612.04858. 2016.
[33] Kulkarni U, Meena SM, Gurlahosur SV, Mudengudi U. Classification of cultural heritage sites using transfer learning. In: 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM); 2019. pp. 391-397.
[34] Yogatama D, Mann G. Efficient transfer learning method for automatic hyperparameter tuning. In: PMLR Artificial Intelligence and Statistics; 2014. pp. 1077-1085.
[35] Law HC, Zhao P, Chan L, Huang J, Sejdinovic D. Hyperparameter learning via distributional transfer. arXiv preprint arXiv:1810.06305 2018.
[36] Perrone V, Jenatton R, Seeger M, Archambeau C. Scalable hyperparameter transfer learning. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems; 2018. pp. 6846-6856.
[37] Perrone V, Shen H, Seeger M, Archambeau C, Jenatton R. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. arXiv preprint arXiv:1909.12552 2019.
[38] Zela A, Klein A, Falkner S, Hutter F. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.06906 2018.
[39] Ozaki Y, Yano M, Onishi M. Effective hyperparameter optimization using Nelder-Mead method in deep learning. IPSJ Transactions on Computer Vision and Applications 2017; 9 (1): 1-12.
[40] Graham B. Kaggle diabetic retinopathy detection competition report. University of Warwick, 2015.
[41] He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision; 2015. pp. 1026-1034.
[42] Paszke A, Gross S, Massa F, Lerer A, Bradbury J et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 2019; 32: 8026-8037.
[43] Wilson AC, Roelofs R, Stern M, Srebro N, Recht B. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292 2017.
[44] Horváth S, Klein A, Richtárik P, Archambeau C. Hyperparameter Transfer Learning with Adaptive Complexity. In: PMLR International Conference on Artificial Intelligence and Statistics; 2021. pp. 1378-1386.