PPoSS: Planning: Cross-Layer Design for Cost-Effective HPC in the Cloud

Many high-performance computing (HPC) applications of national importance (e.g., nuclear simulations, climate modeling, drug discovery, epidemiology, and finance) process enormous datasets and have significant resource demands and strict performance/accuracy/power constraints. Ever-changing hardware elements (e.g., emerging new compute elements) and systems software (continuous fixes to operating systems, compilers and runtime systems) make hosting such HPC applications in locally-managed compute platforms increasingly less attractive. A promising alternate approach is to host these applications in the cloud. However, making legacy HPC applications cloud-ready and identifying the best blend of cloud services for a given application are significant challenges that need to be addressed. In this project, a holistic, cross-layer approach is taken to address the problem of securely mounting such HPC applications in the cloud with high efficiency, low cost, and good performance. A key distinguishing aspect of this project is that it combines both compile-time and run-time innovations and makes contributions to both client and cloud-provider sides. This project spans the following five complementary thrusts, all of which are made challenging by the increasing complexity and scale of the HPC applications of interest, and by the complexity of cloud service offerings and application service-level objectives: (i) characterizing HPC application behavior on myriad cloud infrastructural options; (ii) compiler support for HPC application cloudization; (iii) novel programming language support -- Object-as-a-Service (OaaS); (iv) workload placement and scheduling support; and (v) systems software support for PaaS/SaaS on heterogeneous hardware. The ultimate goal of this project is to devise systematic methodologies for mapping HPC applications to different types of services (spanning IaaS, SaaS, FaaS, OaaS) in multi/hybrid-cloud. This research facilitates improvements in the costs of running HPC applications. This project also enables easy transitioning of HPC applications from one cloud to another and provides data for cloud architecture designers to tune their systems better for current and future HPC workloads. In addition to its technical contributions, this project involves various educational and outreach activities as well. In particular, a new graduate curriculum for cloud computing focusing on HPC applications is created and freely disseminated. Finally, the code being developed and experimental results collected are documented and open-sourced. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.