基于Hadoop平台的数据迁移方法研究实现

Research and Realization of Data Migration Tool Based on Hadoop platform

摘要:

随着计算机科学的发展和大数据时代的到来,应用系统已经出现了数据海量化、用户访问高量化的局面,使得企业应用系统的原有关系型数据库(RDBMS)面临承担更大负荷的压力,系统的高性能要求得不到有效满足,对于关系型数据库所面临的问题,Hadoop平台中的HBase数据库可有效解决。以关系型数据库中MySQL数据库及Hadoop平台中分布式数据库HBase数据库为研究基础,应对企业应用数据海量化增长,提出从关系型数据库(MySQL数据库)向分布式数据库(HBase数据库)进行数据迁移的方法,并通过研究HBase数据库存储原理提出从MySQL到HBase的表模式转换原则实现高效数据查询性能的数据迁移方法。最后,将该方法与同类数据迁移工具Sqoop进行比较,证明该方法进行数据迁移的便捷性和在迁移后数据库中进行连接查询的高效性。

Abstract:

With the development of computer science and the advent of big data era, the application system has appeared the situation of mass data and user access. As a result, the original relational database (RDBMS) of enterprise application system is under the burden of greater load, The system"s high-performance requirements can not be met effectively. For the problems faced by the relational database, the HBase database in the Hadoop platform can be effectively solved. Based on the research on the MySQL database in the relational database and the HBase database in the distributed database on the Hadoop platform, this paper proposes a method of data migration from the relational database (MySQL database) to the distributed database (HBase database) based on the massive growth of enterprise application data. And through learning HBase database storage principle, proposing the table mode conversion principle from MySQL to HBase which could achieve efficient data query performance on data migration. Finally, the method is compared with Sqoop, a similar data migration tool, to demonstrate the ease with which the method can be migrated and the efficiency of the connection queries in the migrated database.