With the rapid development of robot and machine vision, the vision based intelligent robot has been widely used. Machine vision improves the adaptability of the robot control system to the environment, but the degree of adaptation is influenced by the surrounding environment to the machine vision algorithm. In view of this situation, this paper presents an improved robot visual servo control algorithm in machine vision system. This paper proposed an adaptive environment of inter frame difference method of robot visual servo control method based on improved adaptive degree of environmental control system, improving the control precision of the robot. The experimental and theoretical analysis that this method has great application prospect and practical value.