Kiran Kumar EEPURI, Suman MALOJI, Venkata Vijay Kishore POLURIE, Sunitha RAVI

Sign language recognition with multi feature fusion and ANN classifier

Extracting and recognizing complex human movements such as sign language gestures from video sequencesis a challenging task. In this paper this kind of a difficult problem is approached with Indian sign language (ISL) videos.A new segmentation algorithm is developed by fusion of features from discrete wavelet transform (DWT) and local binarypattern (LBP). A 2D point cloud is formed from fused features, which represent the local hand shapes in consecutivevideo frames. We validate the proposed feature extraction model with state of the art features such as HOG, SIFTand SURF for each sign video on the same ANN classifier. We found that the Haar-LBP fused features represent signvideo data in better manner compared to HOG, SIFT and SURF. This is due to the combination of global and localfeatures in our proposed feature matrix. The extracted features input the artificial neural network (ANN) classifier withlabels forming the corresponding words. The proposed ANN classifier is tested against state of the art classifiers such asAdaboost, support vector machine (SVM) and other ANN methods on different features extracted from the ISL dataset.The classifiers were tested for accuracy and correctness in identifying the signs. The ANN classifier that produced arecognition rate of 92.79% was obtained with maximum training instances, which was far greater than the existing workson sign language with other features and ANN classifier on our ISL dataset

PDF

___

Mitra S, Acharya T. Gesture recognition: A survey. IEEE T Syst Man Cy C. 2007; 37: 311-324.
Chen FS, Fu CM, Huang CL. Hand gesture recognition using a real-time tracking method and hidden Markov models. Image Vision Comput 2003; 21: 745-758.
Kushwah MS, Sharma M, Jain K, Chopra A. Sign language interpretation using pseudo glove. In: 1st International Conference on Intelligent Communication, Control and Devices; 2017; Dehradun, India. pp. 9-18.
Kumar P, Gauba H, Roy PP, Dogra DP. Coupled hmm-based multi-sensor data fusion for sign language recognition. Pattern Recogn Lett 2017; 86: 1-8.
Mapari RB, Kharat G. American static signs recognition using leap motion sensor. In: Proceedings of the 2nd International Conference on Information and Communication Technology for Competitive Strategies; 4 March 2016; India. pp. 67.
Auti A, Amolic R, Bharne S, Raina A, Gaikwad DP. Sign-Talk: Hand gesture recognition system. Int J Comput Appl 2017; 160.
Belgacem S, Chatelain C, Paquet T. Gesture sequence recognition with one shot learned CRF/HMM hybrid model. Image Vision Comput 2017; 61: 12-21.
Sun S, Luo C, Chen J. A review of natural language processing techniques for opinion mining systems. Inform Fusion 2017; 36: 10-25.
Indian Sign language, empowering the deaf, http://www.deafsigns.org.
Akmeliawati R, Ooi MP, Kuang YC. Real-time Malaysian sign language translation using colour segmentation and neural network. In: IEEE Instrumentation and Measurement Technology Conference Proceedings; 2007; Warsaw, Poland. pp.1-6.
Zhou Y, Chen X, Zhao D, Yao H, Gao W. Adaptive sign language recognition with exemplar extraction and MAP/IVFS. IEEE Signal Proc Let. 2010; 17: 297-300.
Kishore PVV, Kumar PR. A video based Indian sign Language recognition system (INSLR) using wavelet transform and fuzzy logic. International Journal of Engineering and Technology 2012; 4: 537.
Inthiyaz S, Madhav BT, Kishore PV. Flower segmentation with level sets evolution controlled by colour, texture and shape features. Cogent Engineering 2017; 4: 1323572.
Shimada M, Iwasaki S, Asakura T. Finger spelling recognition using neural network with pattern recognition model. In: IEEE SICE 2003 Annual Conference; 2003; Fukui, Japan. pp. 2458-2463.
Lu H, Setiono R, Liu H. Effective data mining using neural networks. IEEE T Knowl Data En 1996; 8: 957-61.
Rätsch G, Onoda T, Müller KR. Soft margins for AdaBoost. Mach Learn 2001; 42: 287-320.
Wu B, Ai H, Huang C, Lao S. Fast rotation invariant multi-view face detection based on real adaboost. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition; 2004; Seoul, South Korea. pp. 79-84.
Patel CI, Garg S, Zaveri T, Banerjee A, Patel R. Human action recognition using fusion of features for unconstrained video sequences. Computers and Electrical Engineering 2016.
Wang J, She M, Nahavandi S, Kouzani A. A review of vision-based gait recognition methods for human identification. In: IEEE Digital Image Computing: Techniques and Applications Conference; 2010; Sydney, Australia. pp. 320-327.
Ben-Chen M, Gotsman C. Characterizing shape using conformal factors. In: 3DOR; 2008; Crete, Greece. pp. 1-8.
Serhatlıoğlu S, Hardalaç F, Kutbay U, Kocaöz Ö. Analyses of a cirrhotic patient’s evolution using self organizing mapping and Child-Pugh scoring. J Med Syst 2015; 39:17.
P. V. V. Kishore, M. V. D. Prasad, C. R. Prasad and R. Rahul. 4-Camera model for sign language recognition using elliptical fourier descriptors and ANN. In: 2015 International Conference in Signal Processing And Communication Engineering Systems (SPACES); 2015. pp. 34-38