基于MapReduce的贝叶斯网络参数学习方法

Bayesian Network Parameter Learning using MapReduce Based on MapReduce

摘要:

期望最大化(Expectation Maximization,EM)算法常被应用于贝叶斯网络参数学习过程,但在处理海量数据时由于迭代计算过程的复杂性和处理器、内存等资源的限制,该算法的效能受到极大影响；通过对大数据环境下传统线性贝叶斯网络参数学习方法计算复杂性瓶颈问题的研究,提出了基于MapReduce平台的贝叶斯网络并行期望最大化(Parallel Expectation Maximization,PEM)参数学习算法；利用不完备训练样本集,对态势评估贝叶斯网络进行参数学习；仿真结果表明:在大数据条件下PEM算法能够准确的学习网络参数,同时有效减少参数学习所需时间且具有较好的可拓展性。

Abstract:

Expectation maximization (EM) algorithm is usually used in the parameter learning process of Bayesian networks. However, the performance of this algorithm is restricted by the numerical iterative complexity and CPU, RAM. As for the numerical bottleneck when using traditional Expectation Maximization algorithm, the parallel expectation maximization (PEM) algorithm based on MapReduce is proposed. Situation evaluation Bayesian network is built to verify PEM algorithm. The results have shown this algorithm has good precision, efficiency and scalability in big data processing.