Alessandra BANDRABUR, Laura FLOREA, Cornel FLOREA, Matei MANCAS

Late fusion of facial dynamics for automatic expression recognition

Installment of a facial expression is associated with contractions and extensions of specific facial muscles. Noting that expression is about changes, we present a model for expression classification based on facial landmarks dynamics. Our model isolates the trajectory of facial fiducial points by wrapping them up in relevant features and discriminating among various alternatives with a machine learning classification system. The used features are geometric and temporal-based and the classification system is represented by a late fusion framework that combines several neural networks with binary responses. The proposed method is robust, being able to handle complex expression classes.

PDF

___

[1] Pentland A, Heibeck T. Honest Signals: How They Shape Our World. Cambridge, MA, USA: MIT Press, 2010.
[2] Yang Q, Li C, Li Z. Application of FTGSVM algorithm in expression recognition of fatigue driving. Journal of Multimedia 2014; 9: 527-533.
[3] Basu N, Nag S, Bandyopadhyay SK. Retrieval of facial expressions for facilitating crime investigation. Asian J Sci Technol 2016; 7: 2381-2387.
[4] Hazlett RL. Measuring emotional valence during interactive experiences: boys at video game play. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2006. pp. 1023-1026.
[5] Girard J, Cohn J, Mahoor M, Mavadati S, Hammal Z, Rosenwald D. Nonverbal social withdrawal in depression: evidence from manual and automatic analyses. Image Vision Comput 2014; 32: 641-647.
[6] Florea C, Florea L, Vertan C. Learning pain from emotion: transferred hot data representation for pain intensity estimation. In: European Conference on Computer Vision; 2014. pp. 778-790.
[7] Ekman P, Friesen W. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
[8] Valstar MF, Pantic M. Fully automatic recognition of the temporal phases of facial actions. IEEE T Syst Man Cy B 2012; 42: 28-43.
[9] Liu P, Han S, Meng Z, Tong Y. Facial expression recognition via a boosted deep belief network. In: IEEE Conference on Computer Vision and Pattern Recognition; 2014. pp. 1805-1812.
[10] Mery D, Bowyer K. Recognition of facial attributes using adaptive sparse representations of random patches. In: European Conference on Computer Vision; 2014. pp. 778-792.
[11] Williams A. Facial expression of pain: an evolutionary account. Behav Brain Sci 2002; 25: 475-480.
[12] Ekman P, Rosenberg EL. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System. Oxford, UK: Oxford University Press, 1997.
[13] Ekman P, Cordaro D. What is meant by calling emotions basic. Emotion Review 2011; 3: 364-370.
[14] Zeng Z, Pantic M, Roisman GI, Huang TS. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE T Pattern Anal 2009; 31: 39-58.
[15] De la Torre F, Cohn JF. Facial expression analysis. In: Moeslund TB, Hilton A, Kr¨uger V, Sigal L. Visual Analysis of Humans. Berlin, Germany: Springer, 2011. pp. 377-409.
[16] Pantic M, Patras I. Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE T Syst Man Cy B 2006; 36: 433-449.
[17] Bandrabur A, Florea L, Florea C, Mancas M. Emotion identification by facial landmarks dynamics analysis. In: IEEE International Conference on Intelligent Computer Communication and Processing; 2015. pp. 379-382.
[18] Tian YL. Evaluation of face resolution for expression analysis. In: Computer Vision and Pattern Recognition Workshop; 2004. p. 82.
[19] Littlewort G, Bartlett MS, Fasel I, Susskind J, Movellan J. Dynamics of facial expression extracted automatically from video. Image Vision Comput 2006; 24: 615-625.
[20] Jiang B, Valstar MF, Pantic M. Action unit detection using sparse appearance descriptors in space-time video volumes. In: Automatic Face and Gesture Recognition and Workshops; 2011. pp. 314-321.
[21] Zhao G, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE T Pattern Anal 2007; 29: 915-928.
[22] Rivera AR, Castillo JR, Chae OO. Local directional number pattern for face analysis: face and expression recognition. IEEE T Image Process 2013; 22: 1740-1752.
[23] Rudovic O, Pavlovic V, Pantic M. Multi-output Laplacian dynamic ordinal regression for facial expression recognition and intensity estimation. In: Computer Vision and Pattern Recognition Conference; 2012. pp. 2634-2641.
[24] Youssif AA, Asker WAA. Automatic facial expression recognition system based on geometric and appearance features. Stud Comp Intell 2011; 4: 115.
[25] Yi J, Mao X, Chen L, Xue Y, Compare A. Facial expression recognition considering individual differences in facial structure and texture. IET Comput Vis 2014; 8: 429-440.
[26] Shbib R, Zhou S. Facial expression analysis using active shape model. International Journal of Signal Processing, Image Processing and Pattern Recognition 2015; 8: 9-22.
[27] Liu M, Li S, Shan S, Chen X. AU-aware deep networks for facial expression recognition. In: Automatic Face and Gesture Recognition International Conference and Workshops; 2013. pp. 1-6.
[28] Zhong L, Liu Q, Yang P, Liu B, Huang J, Metaxas DN. Learning active facial patches for expression analysis. In: Computer Vision and Pattern Recognition Conference; 2012. pp. 2562-2569.
[29] Kanade T, Cohn J, Tian Y. Comprehensive database for facial expression analysis. In: Automatic Face and Gesture Recognition Proceedings; 2000. pp. 46-53.
[30] Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I. The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops; 2010. pp. 94-101.
[31] Viola P, Jones M. Robust real-time face detection. Int J Comput Vision 2004; 57: 137-154.
[32] Lindeberg T. Scale-space theory: a basic tool for analysing structures at different scales. J Appl Stat 1994; 21: 225-270.
[33] Snoek CG, Worring M, Smeulders AW. Early versus late fusion in semantic video analysis. In: Proceedings of the 13th Annual ACM International Conference on Multimedia; 2005. pp. 399-402.
[34] Tian YL, Bolle, RM. Automatic detecting neutral face for face authentication and facial expression analysis. In: AAAI-03 Spring Symposium on Intelligent Multimedia Knowledge Management; 2003. pp. 24-26.
[35] Chang CC, Lin CJ. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2011; 2: 27.
[36] Zhu X, Ramanan D. Face detection, pose estimation, and landmark localization in the wild. InL Computer Vision and Pattern Recognition Conference; 2012. pp. 2879-2886.
[37] Google Cloud Vision API. Image Content Analysis, Google Cloud Platform.
[38] Cristinacce D, Cootes TF. Feature detection and tracking with constrained local models. In: BMVC; 2006. p. 3.