In order to better apply the existing depth convolution neural network to facial expression recognition, a method combining pre-training of natural facial expression image set and multi-task depth learning is proposed. Firstly, a data set of spontaneous facial expressions is constructed from social network images, and the existing deep convolution neural network is pre-trained. Then, a deep multi-task facial expression model is constructed by replacing the planar softmax classifier of the output layer with a double-tree classifier. The experimental results show that the proposed method can effectively improve the accuracy of facial expression recognition.