TY - GEN
T1 - Scheduling distributed resources in heterogeneous private clouds
AU - Kesidis, George
AU - Shan, Yuquan
AU - Jain, Aman
AU - Urgaonkar, Bhuvan
AU - Khamse-Ashari, Jalal
AU - Lambadaris, Ioannis
N1 - Publisher Copyright:
© 2018 IEEE.
PY - 2018/11/7
Y1 - 2018/11/7
N2 - We first consider the static problem of allocating resources to (i.e., scheduling) multiple distributed application frameworks, possibly with different priorities and server preferences, in a private cloud with heterogeneous servers. Several fair scheduling mechanisms have been proposed for this purpose. We extend prior results on max-min fair (MMF) and proportional fair (PF) scheduling to this constrained multiresource and multiserver case for generic fair scheduling criteria. The task efficiencies (a metric related to proportional fairness) of max-min fair allocations found by progressive filling are compared by illustrative examples. In the second part of this paper, we consider the online problem (with framework churn) by implementing variants of these schedulers in Apache Mesos using progressive filling to dynamically approximate max-min fair allocations. We evaluate the implemented schedulers in terms of overall execution time of realistic distributed Spark workloads. Our experiments show that resource efficiency is improved and execution times are reduced when the scheduler is 'server specific' or when it leverages characterized required resources of the workloads (when known).
AB - We first consider the static problem of allocating resources to (i.e., scheduling) multiple distributed application frameworks, possibly with different priorities and server preferences, in a private cloud with heterogeneous servers. Several fair scheduling mechanisms have been proposed for this purpose. We extend prior results on max-min fair (MMF) and proportional fair (PF) scheduling to this constrained multiresource and multiserver case for generic fair scheduling criteria. The task efficiencies (a metric related to proportional fairness) of max-min fair allocations found by progressive filling are compared by illustrative examples. In the second part of this paper, we consider the online problem (with framework churn) by implementing variants of these schedulers in Apache Mesos using progressive filling to dynamically approximate max-min fair allocations. We evaluate the implemented schedulers in terms of overall execution time of realistic distributed Spark workloads. Our experiments show that resource efficiency is improved and execution times are reduced when the scheduler is 'server specific' or when it leverages characterized required resources of the workloads (when known).
UR - http://www.scopus.com/inward/record.url?scp=85058313157&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85058313157&partnerID=8YFLogxK
U2 - 10.1109/MASCOTS.2018.00018
DO - 10.1109/MASCOTS.2018.00018
M3 - Conference contribution
AN - SCOPUS:85058313157
T3 - Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018
SP - 102
EP - 108
BT - Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018
Y2 - 25 September 2018 through 28 September 2018
ER -
