A random subspace based conic functions ensemble classifier

Classifiers overfit when the data dimensionality ratio to the number of samples is high in a dataset. This problem makes a classification model unreliable. When the overfitting problem occurs, one can achieve high accuracy in the training; however, test accuracy occurs significantly less than training accuracy. The random subspace method is a practical approach to overcome the overfitting problem. In random subspace methods, the classification algorithm selects a random subset of the features and trains a classifier function trained with the selected features. The classification algorithm repeats the process multiple times, and eventually obtains an ensemble of classifier functions. Conic functions based classifiers achieve high performance in the literature; however, these classifiers cannot overcome the overfitting problem when it is the case data dimensionality ratio to the number of samples is high. The proposed method fills the gap in the conic functions classifiers related literature. In this study, we combine the random subspace method and a novel conic function based classifier algorithm. We present the computational results by comparing the new approach with a wide range of models in the literature. The proposed method achieves better results than the previous implementations of conic function based classifiers and can compete with the other well-known methods

PDF

___

[1] Wang H, Cimen E, Singh N, Buckler E. Deep learning for plant genomics and crop improvement. Current Opinion in Plant Biology 2020; 54: 34-41.
[2] Aggarwal C C. Data Mining: The Textbook. New York, USA: Springer Publishing Company, 2015.
[3] Roffo G, Melzi S, Cristani M. Infinite feature selection. In: 2015 IEEE International Conference on Computer Vision (ICCV); Santiago, Chile; 2015. pp. 4202-4210.
[4] Roffo G, Melzi S. Ranking to learn: Feature ranking and selection via eigenvector centrality. In: Appice A, Ceci M, Loglisci C, Masciari E, Ras ZW (editors). New Frontiers in Mining Complex Patterns. Cham, Switzerland: Springer International Publishing, 2016.
[5] Zhang Y, Wang S, Phillips P, Ji G. Binary pso with mutation operator for feature selection using decision tree applied to spam detection. Knowledge-Based Systems 2014; 64: 22-31.
[6] Xuan P, Guo MZ, Wang J, Wang CY, Liu XY et al. Genetic algorithm-based efficient feature selection for classification of pre-mirnas. Genetics and Molecular Research: GMR 2011; 10 (2): 588-603.
[7] Blum A L, Langley P. Selection of relevant features and examples in machine learning. Artificial Intelligence 1997; 97 (1): 245-271.
[8] Chandrashekar G, Sahin F. A survey on feature selection methods. Computers & Electrical Engineering 2014; 40 (1): 16-28.
[9] FRS K P. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 1901; 2 (11): 559-572
10] Golub G H, Reinsch C. Singular Value Decomposition and Least Squares Solutions. Berlin-Heidelberg, Germany: Springer, 1971.
[11] Dumais S T. Latent semantic analysis. Annual Review of Information Science and Technology 2004; 38 (1): 188-230.
[12] Fisher R A. The use of multiple measurements in taxonomic problems. Annals of Eugenics 1936; 7 (2): 179-188.
[13] Vincent P, Larochelle H, Bengio Y, Manzagol P A. Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, ICML ’08; New York, NY, USA; 2008; pp. 1096-1103.
[14] Khalid S, Khalil T, Nasreen S. A survey of feature selection and feature extraction techniques in machine learning. In: 2014 Science and Information Conference; London, UK; 2014. pp. 372-378.
[15] Ho T K. The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis & Machine Intelligence 1998; 20: 832-844.
[16] Ho T K. Random decision forests. In: Proceedings of 3rd International Conference on Document Analysis and Recognition; Quebec, Canada; 1995. pp. 278-282.
[17] Breiman L. Random forests. Machine Learning 2001; 45 (1): 5-32.
[18] Freund Y, Schapire R E. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 1997; 55 (1): 119-139.
[19] Xiong J, Zhang K, Zhang H. A vibrating mechanism to prevent neural networks from overfitting. In: 2019 15th International Wireless Communications Mobile Computing Conference (IWCMC); Tangier, Morocco; 2019. pp. 1737-1742.
[20] Werpachowski R, György A, Szepesvari C. Detecting overfitting via adversarial examples. In: Advances in Neural Information Processing Systems (NIPS 2019); Vancouver, Canada; 2019. pp. 7856-7866.
[21] Feldman V, Frostig R, Hardt M. The advantages of multiple classes for reducing overfitting from test set reuse. arXiv.org 2019; arXiv:1905.10360.
[22] Wu B, Liu Z, Yuan Z, Sun G, Wu C. Reducing overfitting in deep convolutional neural networks using redundancy regularizer. In: Artificial Neural Networks and Machine Learning–ICANN 2017; Alghero, Italy; 2017. pp. 49-55.
[23] Salman S, Liu X. Overfitting mechanism and avoidance in deep neural networks. arXiv.org 2019; arXiv:1901.06566.
[24] Gasimov R N, Ozturk G. Separation via polyhedral conic functions. Optimization Methods and Software 2006; 21 (4): 527-540.
[25] Ozturk G. A new mathematical programming approach to solve classification problems. PhD, Eskişehir Osmangazi University, Institute of Science, Eskişehir, Turkey, 2007. (in Turkish).
[26] Bagirov A M, Ugon J, Webb D, Ozturk G, Kasimbeyli R. A novel piecewise linear classifier based on polyhedral conic and max–min separabilities. TOP 2013; 21 (1): 3-24.
[27] Ozturk G, Bagirov A M, Kasimbeyli R. An incremental piecewise linear classifier based on polyhedral conic separation. Machine Learning 2015; 101 (1): 397-413.
[28] Ozturk G, Ciftci M T. Clustering based polyhedral conic functions algorithm in classification. Journal of Industrial and Management Optimization 2015; 11 (3): 921-932.
[29] Cimen E, Ozturk G. Arrhythmia classification via k-means based polyhedral conic functions algorithm. In: 2016 International Conference on Computational Science and Computational Intelligence; Las Vegas, USA; 2016. pp. 798-802.
[30] Cevikalp H, Triggs B. Polyhedral conic classifiers for visual object detection and classification. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Hawaii, USA, 2017. pp. 4114-4122.
[31] Cimen E. Gesture Recognition with Polyhedral Conic Functions based Classifiers. MS, Graduate School of Science, Anadolu University, Eskişehir, Turkey, 2013 (in Turkish)
32] Uylas Sati N. A binary classification approach based on support vector machines via polyhedral conic functions. Celal Bayar University Journal of Science 2016; 12 (2): 135-149.
[33] Uylas Sati N, Ordin B. Application of the polyhedral conic functions method in the text classification and comparative analysis. Scientific Programming 2018; 2018: 1-11.
[34] Cimen E, Ozturk G, Gerek O N. Incremental conic functions algorithm for large scale classification problems. Digital Signal Processing 2018; 77: 187-194.
[35] Cimen E, Ozturk G, Gerek O N. Icf: An algorithm for large scale classification with conic functions. SoftwareX 2018; 8: 59-63.
[36] Cimen E, Ozturk G. O-pcf algorithm for one-class classification. Optimization Methods and Software 2019; 0 (0): 1-15.
[37] Ozturk G, Cimen E. Polyhedral conic kernel-like functions for svms. Turkish Journal of Electrical Engineering & Computer Sciences 2019; 27: 1172-1180.
[38] Cimen E. Optimization based predictive methods for large scale data. PhD, Eskişehir Technical University, Graduate School of Sciences, Eskişehir, Turkey, 2018.
[39] Dordinejad G G, Cevikalp H. Cone vertex estimation in polyhedral conic classifiers. In: 25th Signal Processing and Communications Applications Conference (SIU); Antalya, Turkey; 2017. pp. 1-4.
[40] Ozturk G, Ceylan G. Max margin polyhedral conic function classifier. In: 2016 International Conference on Computational Science and Computational Intelligence (CSCI); Las Vegas, USA; 2016. pp. 1395-1396.
[41] Zhu J, Rosset S, Hastie T, Tibshirani R. 1normm support vector machines. In: Proceedings of the 16th International Conference on Neural Information Processing Systems, NIPS’03; Vancouver, Canada; 2003. pp. 49-56.
[42] Cortes C, Vapnik V. Support-vector networks. Machine Learning 1995; 20 (3): 273-297.
[43] Zhu J, Zou H, Rosset S, Hastie T. Multi-class adaboost. Statistics and Its Interface 2009; 2: 349-360.
[44] Huang G, Zhou H, Ding X, Zhang R. Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 2012; 42 (2): 513-529.
[45] Gurobi Optimization I. Gurobi optimizer reference manual, 2016.
[46] Lichman M. UCI machine learning repository, 2013.
[47] Zarchi M, Bushehri S F, Dehghanizadeh M. Scadi: A standard dataset for self-care problems classification of children with physical and motor disability. International Journal of Medical Informatics 2018; 114: 81-87.
[48] Guvenir H A, Acar B, Demiroz G, Cekin A. A supervised machine learning algorithm for arrhythmia analysis. In: Computers in Cardiology; Lund, Sweden; 1997. pp. 433-436.
[49] Tsanas A, Little M A, Fox C, Ramig L O. Objective automatic assessment of rehabilitative speech treatment in parkinson’s disease. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2014; 22 (1): 181-190.
[50] Sakar C O, Serbes G, Gunduz A, Tunc HC, Nizam H et al. A comparative analysis of speech signal processing algorithms for parkinson’s disease classification and the use of the tunable q-factor wavelet transform. Applied Soft Computing 2019; 74: 255-263.
[51] Ciarelli P M, Oliveira E. Agglomeration and elimination of terms for dimensionality reduction. In: Proceedings of the 2009 Ninth International Conference on Intelligent Systems Design and Applications, ISDA ’09; Washington, DC, USA; 2009. pp. 547-552.
[52] Ciarelli P M, Salles E O T, Oliveira E. An evolving system based on probabilistic neural network. In: 11th Brazilian Symposium on Neural Networks; Sao Paulo, Brazil; 2010. pp. 182-187.
[53] Kilinc D, Ozcift A, Bozyigit F, Yildirim P, Yucalar F et al. Ttc-3600: A new benchmark dataset for turkish text categorization. Journal of Information Science 2017; 43 (2): 174-185