基于热点数据块的动态副本调整策略

Dynamically Adjusting Strategy of Replica Based on Hot Data Block

摘要:

副本管理策略对于分布式存储系统的可用性、可靠性和系统整体性能有至关重要的作用。本文针对基于文件的动态副本调整策略的不足,提出了一种基于热点数据块的动态副本调整策略。根据时间局部性原理和数据访问规律,通过对历史访问周期和当前周期赋予不同的权重,数据块下一周期的预测进行访问频率计算,接下来基于计算出的预测访问频率对数据块进行热点判定。结合HDFS中数据访问规律近似二八定律的特点和热点数据块的判定结果,来确定数据块的调整阈值。最后,分3个步骤对基于热点数据块的动态副本调整策略进行性详细设计。实验结果表明,本文提出的基于热点数据块的动态副本调整策略在数据访问效率和集群存储资源利用率两方面有了明显提升。

Abstract:

Replica management strategy is critical for the availability, reliability, and overall performance of the distributed storage system. Aimed at the shortage of the dynamic replication strategy based on heat files, a strategy of dynamically adjusting replica factor base on hot data block is proposed. According to the principle of temporal locality and the rule of data access, calculate the predicted access frequency of data block by assigning different weights to the historical visit period and the current period. Then, according to the fact that HDFS data access approximately conform to twenty-eight law to determine the hot data block decision threshold, thereby determining whether a single data block is hot, dynamically adjusting the replication factor of the data block based on the determination result. Finally, the dynamic replica adjustment strategy of hotspot data block is designed in three steps. Experimental results show that the proposed dynamic replica adjustment strategy based on hot data block has been improved in both data access efficiency and cluster storage resource utilization.