This paper explores a novel method to represent face images for facial expression recognition; it is named compact local Gabor directional number pattern (CLGDNP). By convolving the face images with Gabor filters, we encode the magnitude and phase response images in each scale, and calculate the histograms in several nonoverlapping regions of each encoded image. Finally, we obtain two spatial histogram sequences by the aid of the mean pooling technology and concatenate them to form the facial descriptor. Moreover, for evaluating the performance of the proposed method, we employ a support vector machine to conduct some extensive classification experiments on the Radboud faces database, the extended Cohn-Kanade database, and the Japanese Female Facial Expression database. The experimental results demonstrate that the proposed CLGDNP method achieves better performance in classification.