FoMR: Shrinking the Control and Data Flow Latencies of Single Thread Executions for Emerging Workloads

Edge devices such as AR/VR glasses, smart cameras, gaming consoles are permeating rapidly through society for performing numerous applications in surveillance, multi-user gaming, immersive social networking, autonomous control, etc. While one would want to perform all of the data processing at these edge devices, resource constraints with the need for accessing a broader corpus of data, require offloading the computations to a back-end server. These computations are usually parallelized for their throughput needs, but the slowest thread still impacts the latency/responsiveness to the edge device/user. The highly interactive nature of these workloads, thus, makes it imperative to ensure that no thread is left behind, and individual data path of each thread is maximally utilized. This project seeks to address this challenge and work to eliminate the mismatch in responsiveness across various applications run at edge devices. The investigators aim to offer a graduate level course on this subject, help enhance the related undergraduate courses, conduct summer camps for middle school girls.

This project proposes three optimizations each for reducing control and data flow latencies in the normal data path for the emerging edge-serving workloads. The three control flow optimizations include: (i) cross-stack code layouts for better instruction locality, (ii) identifying and boosting the fetch bandwidth of critical instruction chains (CritICs), and (iii) improving branch prediction accuracy especially in shared libraries/OS by leveraging application/data knowledge. On the data flow side, the three optimizations try to leverage the importance of data content: (i) identifying frequent data flow sequences across the entire software stack and memorizing them, (ii) developing content-aware tiling techniques for code generation to reduce re-use distance between same/similar content, and (iii) dynamically leveraging approximations.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.