MROrchestrator: A fine-grained resource orchestration framework for MapReduce clusters

Bikash Sharma, Ramya Prabhakar, Seung Hwan Lim, Mahmut T. Kandemir, Chita R. Das

Research output: Chapter in Book/Report/Conference proceedingConference contribution

37 Scopus citations

Abstract

Efficient resource management in data centers and clouds running large distributed data processing frameworks like MapReduce is crucial for enhancing the performance of hosted applications and increasing resource utilization. However, existing resource scheduling schemes in Hadoop MapReduce allocate resources at the granularity of fixed-size, static portions of nodes, called slots. In this work, we show that MapReduce jobs have widely varying demands for multiple resources, making the static and fixed-size slot-level resource allocation a poor choice both from the performance and resource utilization standpoints. Furthermore, lack of coordination in the management of multiple resources across nodes prevents dynamic slot reconfiguration, and leads to resource contention. Motivated by this, we propose MROrchestrator, a MapReduce resource Orchestrator framework, which can dynamically identify resource bottlenecks, and resolve them through fine-grained, coordinated, and on-demand resource allocations. We have implemented MROrchestrator on two 24-node native and virtualized Hadoop clusters. Experimental results with a suite of representative MapReduce benchmarks demonstrate up to 38% reduction in job completion times, and up to 25% increase in resource utilization. We further demonstrate the performance boost in existing resource managers like NGM and Mesos, when augmented with MROrchestrator.

Original languageEnglish (US)
Title of host publicationProceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012
Pages1-8
Number of pages8
DOIs
StatePublished - 2012
Event2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012 - Honolulu, HI, United States
Duration: Jun 24 2012Jun 29 2012

Publication series

NameProceedings - 2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012

Other

Other2012 IEEE 5th International Conference on Cloud Computing, CLOUD 2012
Country/TerritoryUnited States
CityHonolulu, HI
Period6/24/126/29/12

All Science Journal Classification (ASJC) codes

  • Software

Fingerprint

Dive into the research topics of 'MROrchestrator: A fine-grained resource orchestration framework for MapReduce clusters'. Together they form a unique fingerprint.

Cite this