As to the problem of target image blurring and low tracking accuracy in high-speed missile terminal missile guidance, ballistic experiment and airborne weapon attitude tracking system, a high-speed image tracking system based on high frame rate CMOS image sensor and FPGA is introduced. Image acquisition module, ping-pong high speed cache module, image processing and tracking control module are designed in FPGA by using pipeline technique. Target image is acquired by using adaptive threshold image segmentation algorithm, and the target offset between the centroid and the center of the view field is calculated, pan-tilt is controlled to tracking target according to the offset of two-dimensional view field. Experiments results show that the system can effectively obtain high frame rate image and can read data rate at 80Mb/sec, which improves the system tracking accuracy.