A hybrid approach based on transfer and ensemble learning for improving performances of deep learning models on small datasets

The need for high-volume data is one of the challenging requirements of the deep learning methods, and it makes it harder to apply deep learning algorithms to domains in which the data sources are limited, in other words, small. These domains may vary from medical diagnosis to satellite imaging. The performances of the deep learning methods on small datasets can be improved by the approaches such as data augmentation, ensembling, and transfer learning. In this study, we propose a new approach that utilizes transfer learning and ensemble methods to increase the accuracy rates of convolutional neural networks for classification tasks on small data sets. To this end, we generate different-sized sub-networks by fragmenting an existing large pre-trained network then gather those networks to form an ensemble. For ensemble scoring, we also suggest two new methods. Conducted experiments with the proposed technique, on a randomly sampled Cifar10 small subset dataset, reveals promising results.

PDF

___

[1] McLaughlin N, Del Rincon JM, Miller P. Data-augmentation for reducing dataset bias in person re-identification. In 12th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) 2015: 1-6.
[2] Krizhevsky A, Sutskever I, Hinton G. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing systems, 2012; (25): 1097-1105.
[3] Dieleman S, Willett K, Dambre J. Rotation-invariant convolutional neural networks for galaxy morphology prediction. Monthly Notices of the Royal Astronomical Society, 2015; 450 (2): 1441-1459.
[4] Szegedy C, Liu W, Jia Y,Sermanet P, Reed S et. al. Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 1-9.
[5] Shin HC, Roth HR, Gao M, Lu L, Xu Z et al. Deep Convolutional Neural Networks for Computer-Aided Detection: CNN architectures, dataset characteristics and transfer Learning. IEEE Transactions on Medical Imaging 2016; 35 (5): 1285-1298. doi: 10.1109/TMI.2016.2528162
[6] Razavian A, Azizpour H, Sullivan J, Carlsson S. CNN features off-the-shelf: An astounding baseline for recognition. IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014: 512-519. doi: 10.1109/CVPRW.2014.131
[7] Kandaswamy C, Silva LM, Alexandre LA, Santos JM. Deep transfer learning ensemble for classification. Advances in Computational Intelligence. IWANN 2015. Lecture Notes in Computer Science, Springer, Cham, 2015; 9094: 335-348. doi: 10.1007/978-3-319-19258-1_29
[8] George D, Shen H, Huerta EA. Deep Transfer Learning: A new deep learning glitch classification method for advanced LIGO. ArXiv abs/1706.07446, 2017. doi: 10.1103/PhysRevD.97.101501
[9] Yu Y, Lin H, Meng J, Wei X, Guo H et al. Deep transfer learning for modality classification of medical images. Information, 2017; 8 (3): 91. doi: 10.3390/info8030091
[10] Maji D, Santara A, Mitra P, Sheet D, Ensemble of deep convolutional neural networks for learning to detect retinal vessels in fundus images. ArXiv abs/1603.04833, 2016.
[11] Yan Z, Zhang H, Piramuthu R, Jagadeesh V, DeCoste D et al. HD-CNN: Hierarchical deep convolutional neural network for large scale visual recognition. IEEE International Conference on Computer Vision (ICCV), 2015: 2740- 2748. doi: 10.1109/ICCV.2015.314
[12] Pouyanfar S, Chen S. Semantic event detection using ensemble deep learning. IEEE International Symposium on Multimedia (ISM), 2016: 203-208. doi: 10.1109/ISM.2016.0048.
[13] Ng HW, Nguyen VD, Vonikakis V, Winkler S. Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning. ICMI ’15 Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015: 443-449. doi: 10.1145/2818346.2830593
[14] Korzh O, Joaristi M, Serra E. Convolutional neural network ensemble fine-tuning for extended transfer learning. Big Data – BigData 2018 Lecture Notes in Computer Science, Springer, Cham 2018; 10968: 110-123. doi: 10.1007/978- 3-319-94301-5_9
[15] Nguyen L, Lin D,Lin Z, Cao J. Deep CNNs for microscopic image classification by exploiting transfer learning and feature concatenation. IEEE International Symposium on Circuits and Systems (ISCAS) 2018; 1-5. doi: 10.1109/ISCAS.2018.8351550
[16] Cruz RMO, Sabourin R, Cavalcanti GDC, Ren TI. META-DES: A dynamic ensemble selection framework using meta-learning. Pattern Recognition, 2015; 48 (5): 1925–1935. doi: 10.1016/j.patcog.2014.12.003
[17] Baker B, Gupta O, Naik N, Raska R. Designing neural network architectures using reinforcement learning. ICLR’17: 5th International Conference on Learning Representations, 2017.
[18] Miikkulainen R, Liang J, Meyerson E, Rawal A, Fink D et al. Evolving deep neural networks. ArXiv abs/1707.07012v3, 2017.
[19] Xie L, Yuille A. Genetic CNN. 2017 IEEE International Conference on Computer Vision (ICCV), 2017: 1388-1397. doi: 10.1109/ICCV.2017.154
[20] Song M, Guo P. Combining Local Binary Patterns for Scene Recognition. Journal of Software, 2014; 9: 203-210. doi: 10.4304/jsw.9.1.203-210
[21] Hu J, Guo P. Spatial local binary patterns for scene image classification. Technologies of Information and Telecommunications (SETIT), 2012: 326-330. doi: 10.1109/SETIT.2012.6481936
[22] Shakerdonyavi M, Shanbehzadeh J. Sarrafzadeh A. Large-Scale image retrieval using local binary patterns and iterative quantization. 2015 International Conference on Digital Image Computing: Techniques and Applications (DICTA), 2015: 1-5. doi: 10.1109/DICTA.2015.7371276
[23] Dalal N, Triggs B. Histograms of oriented gradients for human detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005; 1: 886-893. doi: 10.1109/CVPR.2005.177
[24] Ojala T, Pietikäinen M, Harwood D. Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. Proceedings of 12th International Conference on Pattern Recognition, 1994; 1: 582-585. doi: 10.1109/ICPR.1994.576366
[25] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. ArXiv abs/1409.1556, 2015.
[26] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016: 770-778. doi: 10.1109/CVPR.2016.90
[27] Szegedy C, Vanhoucke V, Ioffe S. Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 2818-2826.
[28] Krizhevsky A,Ilya S, Hinton G. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 2012; 25: 1097-1105.
[29] Szegedy C, Liu W, Jia Y, Sermanet P, Reed S et al. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015: 1-9. doi:10.1109/CVPR.2015.7298594
[30] Hashem S. Optimal linear combinations of neural networks. Neural networks : the official journal of the International Neural Network Society, 1997; 10 (4): 599-614. doi:10.1016/S0893-6080(96)00098-6
[31] Taherkhani A, Cosma G, McGinnity TM. AdaBoost-CNN: An adaptive boosting algorithm for convolutional neural networks to classify multi-class imbalanced datasets using transfer learning. Neurocomputing, 2020; 404: 351-366. doi: 10.1016/j.neucom.2020.03.064
[32] He Z, Shao H, Zhong X, Zhao X. Ensemble transfer CNNs driven by multi-channel signals for fault diagnosis of rotating machinery cross working conditions. Knowledge-Based Systems, 2020; 207: 106396. doi: 10.1016/j.knosys.2020.106396
[33] Swati ZNK, Zhao Q, Kabir M, Ali F, Ali Z et al. Brain tumor classification for MR images using transfer learning and fine-tuning. Computerized Medical Imaging and Graphics 2019; 75: 34-46. doi: 10.1016/j.compmedimag.2019.05.001
[34] Moon WK, Lee Y, Ke H, Lee SH, Huang C et al. Computer‐aided diagnosis of breast ultrasound images using ensemble learning from convolutional neural networks. Computer methods and programs in biomedicine, 2020, 190, 105361. doi: 10.1016/j.cmpb.2020.105361
[35] Sahinbas K, Catak FO. Transfer learning-based convolutional neural network for COVID-19 detection with X-ray images. Data Science for COVID-19, 2021: 451–466. doi:10.1016/B978-0-12-824536-1.00003-4
[36] Li Y, Li K, Liu X, Wang Y, Zhang L. Lithium-ion battery capacity estimation—A pruned convolutional neural network approach assisted with transfer learning. Applied Energy 2021; 285: 116410. doi: 10.1016/j.apenergy.2020.116410
[37] Cooney C, Folli R, Coyle D. Optimizing layers improves CNN generalization and transfer learning for imagined speech decoding from EEG. 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) 2019: 1311-1316. doi: 10.1109/SMC.2019.8914246
[38] Chandrarathne G, Thanikasalam K, Pinidiyaarachchi A. A Comprehensive Study on Deep Image Classification with Small Datasets. In: Zakaria Z, Ahmad R. (eds) Advances in Electronics Engineering. Lecture Notes in Electrical Engineering, Springer, Singapore 2020; 619: 93-106.