Aiming at solving the problem that SIFT (scale invariant feature transform) algorithm’s computation complexity is high and its matching accuracy is low in a complex background, an image matching algorithm by combining LBP-HSV model and improved SIFT algorithm is proposed. It first utilizes LBP histogram and HSV model to screen the identified similar region. Then it uses SIFT algorithm to detect the feature points of the target and alternative region, and take advantages of improved HOG feature to describe feature vectors. Finally, it finds matching points by using k-nearest-neighbor algorithm and weighted Euclidean distance. The results of experiments carried out on multiple pedestrian pictures show that the proposed algorithm has good robustness and high accuracy, and compared with SIFT algorithm, the matching speed is greatly improved.