Abstract:Obstacle detection and recognition during navigation is a key challenge for unmanned boat. Since the autonomous cruise of unmanned boat depends on their own accurate detection of their environment, and the ultrasonic or radar technologies commonly used at present have low detection distance and accuracy and weak obstacle avoidance function, so visual methods can be introduced to improve the obstacle avoidance accuracy and used for trajectory generation, localization or path planning. In this paper, stereo vision system is built for unmanned boat, and the 3D space model of the boat is constructed by using stereo Direct Sparse Odometry(DSO) algorithm. Then the 3D point cloud map will be transformed into a 2D grid map and the obstacles will be marked. At last, the obstacle data for obstacle avoidance system is provided to plan path. Experimental results indicate that the system constructs a 2D grid map of the real river environment and solves the problem called "virtual obstacle" in the construction of water environment.