Hadoop平台下基于优化X-means算法的大数据聚类研究

Research on Large Cluster Analysis Based on Optimized X-means Algorithm under Hadoop Platform

摘要:

针对现有聚类方法对数据处理规模的局限性，解决数据聚类效果差的问题，在Hadoop平台的支持下提出基于优化X-means算法的大数据聚类方法。利用Hadoop平台架构与函数采集大数据样本，通过缺失补偿、噪声滤波、归一化等步骤，实现初始样本数据的预处理。选择大数据聚类中心，分别提取聚类中心数据与其他所有数据样本的特征，计算数据样本与聚类中心之间的特征相似度。以相似度度量结果为聚类判定条件，利用优化X-means算法确定数据所属类型，最终实现大数据的聚类处理工作。通过聚类效果测试实验得出结论:在有、无两种实验条件下，与传统聚类方法相比，优化设计方法的查全率和查准率分别提升了4.75%和4.5%，同时优化聚类方法得出数据具有更高利用率。

Abstract:

In response to the limitations of existing clustering methods on data processing scale and the problem of poor data clustering performance, a big data clustering method based on optimized X-means algorithm is proposed with the support of Hadoop platform. Utilizing the Hadoop platform architecture and functions to collect big data samples, preprocessing the initial sample data is achieved through steps such as missing compensation, noise filtering, and normalization. Select a big data clustering center, extract the features of the clustering center data and all other data samples, and calculate the feature similarity between the data samples and the clustering center. Using similarity measurement results as clustering criteria, the optimized X-means algorithm is used to determine the type of data, ultimately achieving clustering processing of big data. Through clustering effectiveness testing experiments, it was concluded that under both experimental conditions, the recall and precision of the optimized design method were improved by 4.75% and 4.5%, respectively, compared to traditional clustering methods. At the same time, the optimized clustering method resulted in higher data utilization.