Article ID | Journal | Published Year | Pages | File Type |
---|---|---|---|---|
720700 | The Journal of China Universities of Posts and Telecommunications | 2013 | 12 Pages |
Abstract
We present an approach to optimize the MapReduce architecture, which could make heterogeneous cloud environment more stable and efficient. Fundamentally different from previous methods, our approach introduces the machine learning technique into MapReduce framework, and dynamically improve MapReduce algorithm according to the statistics result of machine learning. There are three main aspects: learning machine performance, reduce task assignment algorithm based on learning result, and speculative execution optimization mechanism. Furthermore, there are two important features in our approach. First, the MapReduce framework can obtain nodes' performance values in the cluster through machine learning module. And machine learning module will daily calibrate nodes' performance values to make an accurate assessment of cluster performance. Second, with the optimization of tasks assignment algorithm, we can maximize the performance of heterogeneous clusters. According to our evaluation result, the cluster performance could have 19% improvement in current heterogeneous cloud environment, and the stability of cluster has greatly enhanced.
Related Topics
Physical Sciences and Engineering
Engineering
Electrical and Electronic Engineering
Authors
Wen-hui LIN, Zhen-ming LEI, Jun LIU, Jie YANG, Fang LIU, Gang HE, Qin WANG,