异构环境下基于双重预取的Hadoop调度算法

Scheduling Algorithm Based on Double Prefetching in Heterogeneous Hadoop Clusters

摘要:

Hadoop处理海量数据时,无论是Map任务还是Reduce任务都需要耗费大量的时间传输数据,故提出一种基于双重预取的调度算法；该算法通过估算节点上任务执行的进度来预测Map任务的执行节点,然后通知节点提前预取所需的数据,并且在Map任务完成的数量达到预定值时,开始为Reduce任务预取部分数据；由于在异构的环境下集群中节点的性能各不相同,为此采取了改进的预测模型,以提高任务进度判断的准确性；实验证明,本算法在作业响应时间等方面优于现有的调度算法。

Abstract:

When Hadoop processing huge amounts of data, both in the Map tasks and Reduce tasks requires a lot of time to transfer data. This paper presents a scheduling algorithm based on double prefetching, the algorithm predicts the node which will execute the Map task by estimating the progress of running tasks, so that the node can prefetch required data for Map tasks. Moreover, the system can also prefetch the data for Reduce tasks while Map tasks are running. Due to the performance of the cluster nodes in heterogeneous environment are not identical, the algorithm adopts an improved prediction model to improve the accuracy of the judgment of task progress. Experiments show that the algorithm is superior to the existing scheduling algorithm with less response time.