Aim at the contradiction between the calculation efficiency and optimization performance of the existing test sequence optimization algorithm, combining with the discrete particle swarm optimization algorithm, a heuristic evaluation function based on weighted Huffman coding was proposed to improve the traditional AO* algorithm, called DPSO-WAO*(DPSO-Weight_AO*)algorithm. The heuristic evaluation function based on weighted Huffman coding was proved by Examples that it can more accurately evaluate the global test cost. In the case of canceling the cost back, the DPSO-WAO* algorithm can still maintain high optimization performance and effectively reduce the computational complexity of the algorithm. It is of great significance for test sequence design, testability analysis and fault diagnosis of large system.