TY - GEN
T1 - MROrchestrator
T2 - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012
AU - Sharma, Bikash
AU - Prabhakar, Ramya
AU - Lim, Seung Hwan
AU - Kandemir, Mahmut T.
AU - Das, Chita R.
PY - 2012
Y1 - 2012
N2 - Efficient resource management in data centers and clouds running large distributed data processing frameworks like MapReduce is crucial for enhancing the performance of hosted applications and increasing resource utilization. However, existing resource scheduling schemes in Hadoop MapReduce allocate resources at the granularity of fixed-size, static portions of nodes, called slots. In this work, we show that MapReduce jobs have widely varying demands for multiple resources, making the static and fixed-size slot-level resource allocation a poor choice both from the performance and resource utilization standpoints. Furthermore, lack of coordination in the management of multiple resources across nodes prevents dynamic slot reconfiguration, and leads to resource contention. Motivated by this, we propose MROrchestrator, a MapReduce resource Orchestrator framework, which can dynamically identify resource bottlenecks, and resolve them through fine-grained, coordinated, and on-demand resource allocations. We have implemented MROrchestrator on two 24-node native and virtualized Hadoop clusters. Experimental results with a suite of representative MapReduce benchmarks demonstrate up to 38% reduction in job completion times, and up to 25% increase in resource utilization. We further demonstrate the performance boost in existing resource managers like NGM and Mesos, when augmented with MROrchestrator.
AB - Efficient resource management in data centers and clouds running large distributed data processing frameworks like MapReduce is crucial for enhancing the performance of hosted applications and increasing resource utilization. However, existing resource scheduling schemes in Hadoop MapReduce allocate resources at the granularity of fixed-size, static portions of nodes, called slots. In this work, we show that MapReduce jobs have widely varying demands for multiple resources, making the static and fixed-size slot-level resource allocation a poor choice both from the performance and resource utilization standpoints. Furthermore, lack of coordination in the management of multiple resources across nodes prevents dynamic slot reconfiguration, and leads to resource contention. Motivated by this, we propose MROrchestrator, a MapReduce resource Orchestrator framework, which can dynamically identify resource bottlenecks, and resolve them through fine-grained, coordinated, and on-demand resource allocations. We have implemented MROrchestrator on two 24-node native and virtualized Hadoop clusters. Experimental results with a suite of representative MapReduce benchmarks demonstrate up to 38% reduction in job completion times, and up to 25% increase in resource utilization. We further demonstrate the performance boost in existing resource managers like NGM and Mesos, when augmented with MROrchestrator.
UR - http://www.scopus.com/inward/record.url?scp=84866744799&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84866744799&partnerID=8YFLogxK
U2 - 10.1109/CLOUD.2012.37
DO - 10.1109/CLOUD.2012.37
M3 - Conference contribution
AN - SCOPUS:84866744799
SN - 9780769547558
T3 - Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012
SP - 1
EP - 8
BT - Proceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012
Y2 - 24 June 2012 through 29 June 2012
ER -