Foveated HDR: Efficient HDR Content Generation on Edge Devices Leveraging User’s Visual Attention

In recent years, high dynamic range (HDR) content has become increasingly popular for its ability to represent a broader brightness range, enhancing the realism and immersion in applications like augmented reality/virtual reality (AR/VR) on edge devices. While DNN-based solutions are effective for reconstructing high-fidelity HDR content, due to their high computational demands and memory usage, the DNN-based HDR reconstruction takes up to several seconds to generate one HDR image, making it very challenging to deploy such techniques onto the edge devices. Towards this, we propose Foveated HDR for efficient HDR content generation on edge devices by leveraging the “foveation nature” of human vision system and adaptively distributing the compute resources based on visual importance. Unlike the state-of-the-art works which treat all the pixels equally, we prioritize computational resources for the user’s view focus and reduce computation for unimportant regions. Specifically, Foveated HDR does not only dynamically identify the user’s focus of view, but also is a user-driven approach that tailors DNN inference speedup to individual user preferences. Furthermore, Foveated HDR carefully considers the employed DNN model architecture when performing foveated processing to preserve the quality of the generated HDR content. We implement and evaluate the proposed design using an edge GPU SoC (NVIDIA Jetson Orin Nano). The experimental results show that Foveated HDR can speedup the DNN inference by 3.6x, which translates to 75% energy saving, with negligible quality degradation.