Modjtaba ROUHANI, Omid ALMASI NAGHASH

A new fuzzy membership assignment and model selection approach based on dynamic class centers for fuzzy SVM family using the firefly algorithm

The support vector machine (SVM) is a powerful tool for classification problems. Unfortunately, the training phase of the SVM is highly sensitive to noises in the training set. Noises are inevitable in real-world applications. To overcome this problem, the SVM was extended to a fuzzy SVM by assigning an appropriate fuzzy membership to each data point. However, suitable choice of fuzzy memberships and an accurate model selection raise fundamental issues. In this paper, we propose a new method based on optimization methods to simultaneously generate appropriate fuzzy membership and solve the model selection problem for the SVM family in linear/nonlinear and separable/nonseparable classification problems. Both the SVM and least square SVM are included in the study. The fuzzy memberships are built based on dynamic class centers. The firefly algorithm (FA), a recently developed nature-inspired optimization algorithm, provides variation in the position of class centers by changing their attributes values. Hence, adjusting the place of the class center can properly generate accurate fuzzy memberships to cope with both attribute and class noises. Furthermore, through the process of generating fuzzy memberships, the FA can choose the best parameters for the SVM family. A set of experiments is conducted on nine benchmarking data sets of the UCI data base. The experimental results show the effectiveness of the proposed method in comparison to the seven well-known methods of the SVM literature.

PDF

___

[1] Guyon I, Boser B, Vapnik V. Automatic capacity tuning of very large VC-dimension classifier. Adv Neur In 1993; 5: 147-155.
[2] Vapnik VN. Statistical Learning Theory. New York, NY, USA: Wiley, 1998.
[3] Vapnik VN. An overview of statistical learning theory. IEEE T Neural Networ 1999; 10: 988-999.
[4] Burges CJC. A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 1998; 2: 955-974.
[5] Zhang XG. Using class-center vectors to build support vector machines. In: IEEE 1999 Workshop on Signal Processing Society; August 1999; Madison, MI, USA. New York, NY, USA: IEEE. pp. 3-11.
[6] Xiong H, Pandey G, Steinbach M, Kumar V. Enhancing data analysis with noise removal. IEEE T Knowl Data Eng 2006; 18: 304-319.
[7] Hulse JV, Khoshgoftaar TM. Knowledge discovery from imbalanced and noisy data. Data Knowl Eng 2009; 68: 1513-1542.
[8] Mavroforakis ME, Theodoridis S. A geometric approach to support vector machine (SVM) classification. IEEE T Neural Networ 2006; 17: 671-682.
[9] Angelova A, Abu-Mostafa Y, Perona P. Pruning training sets for learning of object categories. In: IEEE 2005 Conference on Computer Vision and Pattern Recognition; 2025 June 2005; San Diego, CA, USA. New York, NY, USA: IEEE. pp. 494-501.
[10] Zhu X, Wu X. Class noise vs. attribute noise: a quantitative study of their impacts. Artif Intell Rev 2004; 22: 177-210.
[11] Brodley CE, Friedl MA. Identifying mislabeled training data. J Artif Intell Res 1999; 11: 131-167.
[12] Hulse JV, Khoshgoftaar TM, Huang H. The pairwise attribute noise detection algorithm. Knowl Inf Syst 2007; 11: 171-190.
[13] Khoshgoftaar TM, Zhong S, Joshi V. Enhancing software quality estimation using ensemble-classifier based noise filtering. Intell Data Anal 2005; 9: 3-27.
[14] Wu X, Zhu X. Mining with noise knowledge: error-aware data mining. IEEE T Syst Man Cy 2008; 38: 917-932.
[15] Lin CF, Wang SD. Fuzzy support vector machines. IEEE T Neural Networ 2002; 13: 464-471.
[16] Lin CF, Wang SD. Training algorithms for fuzzy support vector machines with noisy data. Pattern Recogn Lett 2004; 25: 1647-1656.
[17] Jiang XF, Zhang Y, Lv JC. Fuzzy SVM with a new fuzzy membership function. Neural Comput Appl 2006; 15: 268-276.
[18] Tang WM. Fuzzy SVM with a new fuzzy membership function to solve the two-class problems. Neural Process Lett 2011; 34: 209-219.
[19] Peng X, Wang Y. A geometric method for model selection in support vector machine. Expert Syst Appl 2009; 36: 5745-5749.
[20] Wang S, Meng B. Parameter selection algorithm for support vector machine. Procedia Environ Sci 2011; 11: 538-544.
[21] Chapelle O, Vapnik VN, Bousquet O, Mukherjee S. Choosing multiple parameters for support vector machines. Mach Learn 2002; 46: 131-159.
[22] Opper M, Winther O. Gaussian processes and SVM: mean field and leave-one-out. In: Smola AJ, Bartlett PL, Scholkopf B, Schuurmans D, editors. Advances in Large Margin Classifiers. Cambridge, MA, USA: MIT Press, 2000. pp. 311-326.
[23] Vapnik V, Chapelle O. Bounds on error expectation for support vector machines. Neural Comput 2000; 12: 2013- 2036.
[24] Keerthi SS. Efficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms. IEEE T Neural Networ 2002; 13: 1225-1229.
[25] Sun J, Zheng C, Li X, Zhou Y. Analysis of the distance between two classes for tuning SVM hyperparameters. IEEE T Neural Networ 2010; 21: 305-318.
[26] Guo XC, Yang JH, Wu CG, Wang CY, Liang YC. A novel LS-SVMs hyper-parameter selection based on particle swarm optimization. Neurocomputing 2008; 71: 3211-3215.
[27] Glasmachers T, Igel C. Gradient-based adaptation of general Gaussian kernels. Neural Comput 2005; 17: 2099-2105.
[28] Keerthi SS, Lin CJ. Asymptotic behavior of support vector machines with Gaussian kernel. Neural Comput 2003; 15: 1667-1689.
[29] Wang S, Meng B. PSO algorithm for support vector machine. In: IEEE 2010 Conference on Electronic Commerce and Security; 2931 July 2010; Guangzhou, Hong Kong. New York, NY, USA: IEEE. pp. 163-167.
[30] Lei P, Yi L. Parameter selection of support vector machine using an improved PSO algorithm. In: IEEE 2010 2nd International Conference on Intelligent HumanMachine Systems and Cybernetics; 2628 August 2010; Nanjing, China. New York, NY, USA: IEEE. pp. 221-225.
[31] Lin SW, Ying KC, Chen SC, Lee ZJ. Particle swarm optimization for parameter determination and feature selection of support vector machines. Expert Syst Appl 2008; 35: 1817-1824.
[32] Cheng W, Ding J, Kong W, Chai T, Qin SJ. An adaptive chaotic PSO for parameter optimization and feature extraction of LS-SVM based modeling. In: 2011 American Control Conference; 29 June1 July 2011; San Francisco, CA, USA. New York, NY, USA: IEEE. pp. 3263-3268.
[33] Luo Z, Zhang W, Li Y, Xiang M. SVM parameters tuning with quantum particles swarm optimization. In: IEEE 2008 Conference on Cybernetics and Intelligent Systems; 2124 September 2008; Chengdu, China. New York, NY, USA: IEEE. pp. 324-329.
[34] Zhang W, Niu P. LS-SVM based on chaotic particle swarm optimization with simulated annealing and application. In: IEEE 2011 2nd International Conference on Intelligent Control and Information Processing; 2528 July 2011; Harbin, China. New York, NY, USA: IEEE. pp. 931-935.
[35] Blondin J, Saad A. Metaheuristic techniques for support vector machine model selection. In: IEEE 2010 10th International Conference on Hybrid Intelligent Systems; 2325 August 2010; Atlanta, GA, USA. New York, NY, USA: IEEE. pp. 197-200.
[36] Wu CH, Tzeng GH, Goo YJ, Fang WC. A real-valued genetic algorithm to optimize the parameters of support vector machine for predicting bankruptcy. Expert Syst Appl 2007; 32: 397-408.
[37] Frohlich H, Chapelle O, Scholkopf B. Feature selection for support vector machines by means of genetic algorithms. In: IEEE 2003 15th International Conference on Tools with Artificial Intelligence; 35 November 2013; Sacramento, CA, USA. New York, NY, USA: IEEE. pp. 142-148.
[38] Huang CL, Wang CJ. A GA-based feature selection and parameters optimization for support vector machines. Expert Syst Appl 2006; 31: 231-240.
[39] Lihu A, Holban S¸. Real-valued genetic algorithms with disagreements. In: Pelta D, Krasnogor N, Dumitrescu D, Chira C, Lung R, editors. Nature Inspired Cooperative Strategies for Optimization. Berlin, Germany: Springer, 2012. pp. 333-346.
[40] Yang XS. Nature-Inspired Metaheuristic Algorithms. 1st ed. Bristol, UK: Luniver Press, 2008.
[41] Yang XS. Firefly algorithms for multimodal optimization. In: Watanabe O, Zeugmann T, editors. Stochastic Algorithms: Foundations and Applications. Berlin, Germany: Springer-Verlag, 2009. pp. 169-178.
[42] Lin HT, Lin CJ. A Study on Sigmoid Kernels for SVM and the Training of Non-PSD Kernels by SMO-Type Methods. Taipei, Taiwan: Taiwan University, 2003.
[43] Bordes A, Ertekin S, Weston J, Bottou L. Fast kernel classifiers with online and active learning. J Mach Learn Res 2005; 6: 1579-1619.
[44] Suykens JAK, Gestel TV, De Brabanter J, De Moor B, Vandewalle J. Least Squares Support Vector Machines. Singapore: World Scientific Publishing, 2002.
[45] Suykens JAK, Vandewalle J, De Moor B. Optimal control by least squares support vector machines. Neural Networks 2001; 14: 23-35.
[46] Chuang CC. Fuzzy weighted support vector regression with a fuzzy partition. IEEE T Syst Man Cy 2007; 37: 630-640.
[47] Yu L, Lai K, Wang S, Zhou L. A least squares fuzzy SVM approach to credit risk assessment. In: Cao B, editor. Fuzzy Information and Engineering. Berlin, Germany: Springer-Verlag, 2007. pp. 865-874.
[48] Senthilnath J, Omkar SN, Mani V. Clustering using firefly algorithm: performance study. Swarm Evol Comput 2011; 1: 164-171.
[49] Yang XS, Hosseini SS, Gandomi AH. Firefly algorithm for solving non-convex economic dispatch problems with valve loading effect. Appl Soft Comput 2011; 12: 1180-1186.
[50] Williams P, Li S, Feng J, Wu S. A geometrical method to improve performance of the support vector machine. IEEE T Neural Networ 2007; 18: 942-947.
[51] Ding S, Liu X. Evolutionary computing optimization for parameter determination and feature selection of support vector machines. In: IEEE 2009 Conference on Computational Intelligence and Software Engineering; 1113 December 2009; Wuhan, China. New York, NY, USA: IEEE. pp. 1-5.
[52] Weston J. Leave-one-out support vector machines. In: Dean T, editor. Proceedings of the 16th International Joint Conference on Artificial Intelligence. San Francisco, CA, USA: Morgan Kaufmann, 1999. pp. 727733.
[53] Browne MW. Cross-validation methods. J Math Psychol 2000; 44: 108-132.
[54] Murphy PM, Aha DW. UCI repository of machine learning databases. Irvine, CA, USA: UCI. Available online at www.ics.uci.edu/mlearn/MLRepository.html.
[55] Ratsch G, Onoda T, Meuller K-R. Soft margins for AdaBoost. Mach Learn 2001; 42: 287-320.
[56] Weston J, Herbrich R. Adaptive margin support vector machines. In: Smola A, Bartlett P, Scholkopf B, Schuurmans D, editors. Advances in Large Margin Classifiers. Cambridge, MA, USA: MIT Press, 2000. pp. 281-295.