SHF: Medium: Exploring an Edge Platform Design Trajectory for Next Generation XR Applications

Extended Reality (XR) encompassing virtual reality (VR), where a computer-generated environment can be explored and interacted with by users, at one end of the spectrum and augmented reality (AR), where parts of users' physical world are enhanced with computer-generated data, at the other end of the spectrum, is an emerging technology with transformational potential in many domains of national importance including science, healthcare, education, entertainment, and commerce. While various flavors of XR products (tethered, standalone, screenless) are commercially available today, they only offer point solutions for very specific applications varying in cost, performance and power envelope. Furthermore, as many XR applications will run in an edge-computing scenario, their design demands not only power and energy considerations, but also the adaptability and field-customization to efficiently support any future applications. As a result, there is a critical need to develop frameworks that enable designers to generate, test, evaluate and customize a large spectrum of next-generation XR devices with accompanying software support. Such frameworks can expedite time-to-discovery for AR/VR-based applications and redefine the future of computing. This project seeks to design an agile and cost-effective hardware and software platform, AVATAR, that can be progressively adapted and customized to the performance, quality-of-service (QoS), and energy-efficiency needs of a diverse set of XR applications. AVATAR consists of four inter-twined thrusts. First, a detailed profiling of four types of XR applications on three different edge platforms is conducted to understand the latency, power and utilization of different stages in an XR pipeline for identifying the bottleneck stages/kernels. Second, it undertakes a coordinated design of the compiler and runtime system for controlling tunable knobs such as code partitioning and dynamic reconfiguration. Next, hardware design alternatives embracing device heterogeneity such as general-purpose CPU cores, graphic processing units (GPUs), and field programmable gate arrays (FPGAs) for the targeted kernels/pipeline stages of an XR application along with scope for reconfigurability are explored for desired performance-power tradeoffs. In addition, the scope for exploiting Wi-Fi/5G communication to facilitate efficient computation partitioning between an edge device and cloud is investigated. Finally, a comprehensive evaluation framework to evaluate the efficacy of the proposed solutions for different XR applications is being developed. On the educational front, AVATAR includes a new special-topics course and involvement of undergraduate and graduate students in this emerging research field, where students get exposure to cross-cutting topics in computer architecture, compilers, runtime systems and networking, as well as new classes of XR applications. It also includes several Broadening Participation in Computing (BPC) activities such as a summer camp targeting girls and collaboration with the Education Department at Penn State to expose K-12 students to many areas of computer science and engineering, with the primary focus being on XR applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.