It is very important to improve the fault diagnosis ability to ensure the stable operation of the autonomous underwater vehicle (AUV) system. A fault detection method based on detection is proposed in this paper for AUV propeller system. Firstly, the gray prediction algorithm is used to complete data preprocessing for the data collected by sensors. Secondly, an improved iterative clustering algorithm combining K-mean with is put forward to detect outliers. Finally, the simulation results demonstrate the outlier detection algorithm based on gray prediction and KID has higher accuracy, compared with K-mean and DBSCAN algorithms, which could implement a comprehensive unsupervised fault diagnosis of AUV.