基于云计算Hadoop异构集群的并行作业调度算法

Algorithm for Concurrent Job Scheduling Based on Cloud Computing Hadoop Heterogeneous Computer Cluster[HS)]

摘要:

针对Hadoop异构集群中计算和数据资源的不一致分布所导致的调度性能较低的缺点,设计了一种基于Hadoop集群和改进Late算法的并行作业调度算法;首先,介绍了基于Hadoop框架和Map-Reduce模型的调度原理,然后,在经典的Late调度算法的基础上,对Map任务和Reduce任务的各阶段执行时间进度比例进行存储和更新,为了进一步地提高调度效率,将慢任务迁移到本地化节点或离数据资源较近的物理节点上,并给了基于改进Late算法的作业调度流程;为了验证文中方法,在Hadoop集群系统上测试,设定1个为Jobtracker主控节点和7个为TaskTracker节点,实验结果表明文中方法能实现异构集群的作业调度,且与其它方法比较,具有较低的预测误差和较高的调度效率。

Abstract:

Aiming at the Hadoop heterogeneous computer clusters that have the defects of low scheduling efficiency for in-conformal computing and data resource, a job scheduling method based on Late algorithm based on Hadoop computer cluster was proposed. Firstly, the principle of scheduling based on Hadoop and Map-Reduce model was introduced, then on the basis of the classic scheduling algorithm, the executing time ratio for stages of Map task and Reduce task was stored and renewed, In order to improve the scheduling efficiency further, the slow task is transferred to the local node or the neighbor nodes near the data resource, and the algorism flow for the improved Late algorism was given. In order to verify the method in this paper, the experiment was simulated in the Hadoop computer cluster, the number for Jobtracker nodes and Tasktracker number is 1 and 7, respectively, and the result shows the method in this paper can realize job scheduling for heterogeneous computer clusters, and compared with the other method, it has the low predicating error and high scheduling efficiency.