Scheduling Support for High Performance Clusters

Clusters are becoming a predominant and cost-effective platform

for high performance parallel computing. These systems are gaining acceptance

not just for scientific applications, which have traditionally needed

supercomputers, but are also being deployed to meet the needs of

applications in domains such as databases, web service, multimedia, and

graphics/visualization. Traditional parallel scheduling techniques

such as simple space sharing or coscheduling, which are optimized more

for throughput, may not be able to handle the responsiveness needs of

these emerging applications. At the same time, leaving it entirely

to the native operating system scheduler at each node to make independent

decisions can have a detrimental effect on the performance of communicating

applications. Considering these different issues, this proposal will

explore techniques for efficient management and coordination of the

CPU resources across the nodes of a cluster. This research will develop

a framework to design cluster scheduling mechanisms taking communication

into account together with working sets and I/O information, accomodate

the constraints and idiosyncrasies of an application, and adapt dynamically

to the system load. The mechanisms will be evaluated with real workloads,

using simulation, analytical models and actual implementations.

The outcome of this research will take an important step towards making

clusters more ubiquitous, multiprogrammed, manageable and user-friendly.