基于决策树挖掘算法的气象大数据云平台设计

Design of Meteorological Big Data Cloud Platform Based on Classification And Regression Trees Mining Algorithm

摘要:

大数据、云计算技术的迅猛发展为挖掘气象数据丰富的科研和经济价值提供了技术支撑,促进了Hadoop及其包含的文件存储系统(HDFS,Hadoop Distributed File System)和分布式计算模型在气象数据处理领域广泛应用。由于气象数据具有大数据的4V特征,还需要引入新的数据处理算法来提高气象数据处理效率。通过对决策树算法原理的研究,基于Hadoop云平台,创建随机森林模型,为数据挖掘算法在云平台上的应用提供一种新的可能性。基于决策树(CART,Classification And Regression Trees)挖掘算法的气象大数据云平台设计,采用Hadoop系统架构和MapReduce工作流程,对气象大数据云平台采用集群部署。平台总体架构分为基础设施层、数据管理与处理层、应用层,减少了决策树建立的时间,实现了气象数据高效加工和挖掘分析等平台功能。

Abstract:

The rapid development of big data and cloud computing technology provides technical support for mining the rich scientific research and economic value of meteorological data. It promotes the wide application of Hadoop and Hadoop Distributed File System (HDFS) and distributed computing model in the field of meteorological data processing. Due to the 4V characteristics of big data, new data processing algorithms need to be introduced to improve the efficiency of meteorological data processing. Through the research on the principle of Classification And Regression Trees (CART) algorithm, based on Hadoop cloud platform, a random forest model is created, which provides a new possibility for the application of data mining algorithm on cloud platform. The design of meteorological big data cloud platform based on CART mining algorithm adopts Hadoop system architecture and MapReduce workflow to deploy the meteorological big data cloud platform in clusters. The overall architecture of the platform is divided into infrastructure layer, data management and processing layer, application layer, which reduces the time to establish the decision tree and realizes the big data cloud platform functions such as efficient processing and mining analysis of meteorological data.