The defect detection of towel fabrics was studied in this paper. Through research and analysis, it was found that most of the defects of towel fabrics are mainly concentrated on the characteristics of the towel surface. According to the relationship between the defect and the contour feature of the towel, the defect detection algorithm of towel fabric based on frequency domain analysis was proposed. The energy statistical characteristics of towel fabric image spectrum were used as feature vectors for defect detection. Support Vector Machine (SVM) was used as defect classification and recognition algorithm. Matlab simulation results showed that the average recognition rate of towel fabric defects can reach 95%. The conclusion showed that the energy feature extraction method in frequency domain is simple and effective in defect detection of pure color towel fabric.