基于云环境下的海量大数据存储系统设计

Storage and Management of Massive Data in Cloud Computing

摘要:

云计算是目前国际和国内新兴的一项热门技术,正在给人们的生产生活方式带来深刻的变革；在分布式文件系统中,为了实现数据的可用性和持久性,数据通常被分割成大小相同的文件块,通过多副本的形式分布存储在不同地理位置；大型的云计算服务提供商拥有自己的数据中心,这些数据中心往往分布在世界的不同角落,数据以副本的形式分布在不同的地理位置可以实现更好的容错机制,从而提供高可用的持久化的数据存储；文章主要研究了海量大数据存储系统设计,系统设计以Hadoop为平台,提出了一种新的基于云计算环境的海量大数据存储设计方法,主要给出了文件存储方案设计以及副本方案设计等,为云计算海量数据存储与管理提供了一种可行的关键技术方案。

Abstract:

The explosive growth of enterprise and scientific data poses a huge challenge to the storage and management of massive data. In recent years, as a data intensive computing platform, Hadoop has attracted much attention from both industry and academy. In the distributed file system, in order to availability and persistent data, the data are usually divided into files of the same size block, through the form of distributed memory multi copies are stored in different geographic locations. Large cloud computing service providers have their own data center, different parts of these data centers are often distributed in the world, the data in different geographic locations can achieve fault tolerance mechanism better as a copy distribution, thus providing the persistence of high availability of data storage. Based on the Hadoop platform, this paper proposes a framework of storage and management of massive data, and analyzes key techniques that we need to implement the proposed framework.