In order to improve the accuracy of the microscope inspection system, built up the error model of the inspection system. Factors affecting accuracy of the inspection system were analyzed. Proposed the compensation methods of the movement system and visual system. A motion control mode of open-loop control and position detection was designed, and an edge detection arithmetic based on polynomial fitting was proposed. Results showed that the motion control mode and the edge detection arithmetic could reduce the system error, and detection accuracy was improved.