After existing malware detection algorithm study found that certain detection algorithm can detect only a certain kind of malware and high missing rate in detecting malicious programs. Aiming at the existing detection algorithm lack of comprehensive capacity, we propose a new detection algorithm in this article, the detection algorithm has some integrated detection capabilities. The new algorithm ideas are as follows:The first step is to distinguish between a non-malicious software or software malware, malicious software if its signature is extracted, and then use the decision tree based on malware signatures of malware identification and classification, if signature does not recognize the existence of malware, then according to the characteristics of viruses and worms use similarity calculation algorithm for unknown malware similarity calculation, the final decision system using the algorithm to calculate the similarity of the results of decision-making, the malicious software is a virus or a worm. The similarity calculation algorithm, decision tree and decision-making system in the application of algorithms to detect malicious software is the innovation of this paper.