At present, face recognition methods mainly target at static images. In video surveillance, faces of different video frames are correlated and only part of the faces can effectively reflect face information. A face recognition method based on surveillance video is proposed according to the changing characteristics of face images in surveillance video. Firstly, video face sequence is obtained by combining face detection and tracking technology, and then face images representing all face sequence images are selected for recognition based on partial face image recognition results of video face sequence. Finally, face information is comprehensively reflected according to all face image recognition results selected. Experimental results show that this method can effectively improve the real-time performance of face recognition in video surveillance while ensuring the recognition rate and error rate.