A dynamic data placement policy for heterogeneous Hadoop cluster

Santa Maria Shithil; Tushar Kanti Saha; Tanusree Sharma

doi:10.1109/ICAEE.2017.8255371

A dynamic data placement policy for heterogeneous Hadoop cluster

Santa Maria Shithil, Tushar Kanti Saha, Tanusree Sharma

College of Information Sciences and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

8 Scopus citations

Abstract

Hadoop Distributed File System (HDFS) is a file system for storing and managing big data. The current HDFS block placement policy works well for the homogeneous cluster. However, it can not evenly and fairly distribute blocks across the heterogeneous cluster and results in an unbalanced cluster. An unbalanced cluster also reduces MapReduce performance. Hadoop relies on load balancer tool to balance replica distributions that in turn degrade the overall performance of hadoop. In this paper, we proposed a dynamic block placement policy that distributes blocks across a heterogeneous cluster more evenly than the current HDFS block placement policy and also improves the performance of MapReduce applications. The solution had been implemented and test had been conducted to evaluate its contribution to Hadoop.

Original language	English (US)
Title of host publication	4th International Conference on Advances in Electrical Engineering, ICAEE 2017
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	302-307
Number of pages	6
ISBN (Electronic)	9781538608692
DOIs	https://doi.org/10.1109/ICAEE.2017.8255371
State	Published - Jul 1 2017
Event	4th International Conference on Advances in Electrical Engineering, ICAEE 2017 - Dhaka, Bangladesh Duration: Sep 28 2017 → Sep 30 2017

Publication series

Name	4th International Conference on Advances in Electrical Engineering, ICAEE 2017
Volume	2018-January

Conference

Conference	4th International Conference on Advances in Electrical Engineering, ICAEE 2017
Country/Territory	Bangladesh
City	Dhaka
Period	9/28/17 → 9/30/17

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Computer Science Applications
Biomedical Engineering
Electrical and Electronic Engineering
Instrumentation

Access to Document

10.1109/ICAEE.2017.8255371

Cite this

Shithil, S. M., Saha, T. K., & Sharma, T. (2017). A dynamic data placement policy for heterogeneous Hadoop cluster. In 4th International Conference on Advances in Electrical Engineering, ICAEE 2017 (pp. 302-307). (4th International Conference on Advances in Electrical Engineering, ICAEE 2017; Vol. 2018-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICAEE.2017.8255371

@inproceedings{4fcfe989256d47bdabf3b70076f84430,

title = "A dynamic data placement policy for heterogeneous Hadoop cluster",

abstract = "Hadoop Distributed File System (HDFS) is a file system for storing and managing big data. The current HDFS block placement policy works well for the homogeneous cluster. However, it can not evenly and fairly distribute blocks across the heterogeneous cluster and results in an unbalanced cluster. An unbalanced cluster also reduces MapReduce performance. Hadoop relies on load balancer tool to balance replica distributions that in turn degrade the overall performance of hadoop. In this paper, we proposed a dynamic block placement policy that distributes blocks across a heterogeneous cluster more evenly than the current HDFS block placement policy and also improves the performance of MapReduce applications. The solution had been implemented and test had been conducted to evaluate its contribution to Hadoop.",

author = "Shithil, {Santa Maria} and Saha, {Tushar Kanti} and Tanusree Sharma",

note = "Publisher Copyright: {\textcopyright} 2017 IEEE.; 4th International Conference on Advances in Electrical Engineering, ICAEE 2017 ; Conference date: 28-09-2017 Through 30-09-2017",

year = "2017",

month = jul,

day = "1",

doi = "10.1109/ICAEE.2017.8255371",

language = "English (US)",

series = "4th International Conference on Advances in Electrical Engineering, ICAEE 2017",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "302--307",

booktitle = "4th International Conference on Advances in Electrical Engineering, ICAEE 2017",

address = "United States",

}

Shithil, SM, Saha, TK & Sharma, T 2017, A dynamic data placement policy for heterogeneous Hadoop cluster. in 4th International Conference on Advances in Electrical Engineering, ICAEE 2017. 4th International Conference on Advances in Electrical Engineering, ICAEE 2017, vol. 2018-January, Institute of Electrical and Electronics Engineers Inc., pp. 302-307, 4th International Conference on Advances in Electrical Engineering, ICAEE 2017, Dhaka, Bangladesh, 9/28/17. https://doi.org/10.1109/ICAEE.2017.8255371

A dynamic data placement policy for heterogeneous Hadoop cluster. / Shithil, Santa Maria; Saha, Tushar Kanti; Sharma, Tanusree.
4th International Conference on Advances in Electrical Engineering, ICAEE 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 302-307 (4th International Conference on Advances in Electrical Engineering, ICAEE 2017; Vol. 2018-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A dynamic data placement policy for heterogeneous Hadoop cluster

AU - Shithil, Santa Maria

AU - Saha, Tushar Kanti

AU - Sharma, Tanusree

PY - 2017/7/1

Y1 - 2017/7/1

N2 - Hadoop Distributed File System (HDFS) is a file system for storing and managing big data. The current HDFS block placement policy works well for the homogeneous cluster. However, it can not evenly and fairly distribute blocks across the heterogeneous cluster and results in an unbalanced cluster. An unbalanced cluster also reduces MapReduce performance. Hadoop relies on load balancer tool to balance replica distributions that in turn degrade the overall performance of hadoop. In this paper, we proposed a dynamic block placement policy that distributes blocks across a heterogeneous cluster more evenly than the current HDFS block placement policy and also improves the performance of MapReduce applications. The solution had been implemented and test had been conducted to evaluate its contribution to Hadoop.

AB - Hadoop Distributed File System (HDFS) is a file system for storing and managing big data. The current HDFS block placement policy works well for the homogeneous cluster. However, it can not evenly and fairly distribute blocks across the heterogeneous cluster and results in an unbalanced cluster. An unbalanced cluster also reduces MapReduce performance. Hadoop relies on load balancer tool to balance replica distributions that in turn degrade the overall performance of hadoop. In this paper, we proposed a dynamic block placement policy that distributes blocks across a heterogeneous cluster more evenly than the current HDFS block placement policy and also improves the performance of MapReduce applications. The solution had been implemented and test had been conducted to evaluate its contribution to Hadoop.

UR - http://www.scopus.com/inward/record.url?scp=85047745641&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85047745641&partnerID=8YFLogxK

U2 - 10.1109/ICAEE.2017.8255371

DO - 10.1109/ICAEE.2017.8255371

M3 - Conference contribution

AN - SCOPUS:85047745641

T3 - 4th International Conference on Advances in Electrical Engineering, ICAEE 2017

SP - 302

EP - 307

BT - 4th International Conference on Advances in Electrical Engineering, ICAEE 2017

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 4th International Conference on Advances in Electrical Engineering, ICAEE 2017

Y2 - 28 September 2017 through 30 September 2017

ER -

A dynamic data placement policy for heterogeneous Hadoop cluster

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this