QoS-Diff: Adaptive Auto-tuning Framework for Low-latency Diffusion Model Inference

Diffusion models are pivotal for generating high-quality images, yet they encounter latency and throughput challenges in data center environments, particularly in meeting stringent service level objectives (SLOs).This paper introduces the Quality of Service-Diffusion (QoS-Diff) framework tailored to optimize the diffusion model's performance in data center environments.QoS-Diff dynamically adjusts processing steps per query, enhancing throughput and reducing latency while ensuring SLO compliance.It utilizes latent vector entropy for step adjustment and incorporates a 'one last step' approach to enhance image quality.Additionally, the system integrates load balancing and CPU-GPU pipelining to boost throughput further.Extensive experiments show that QoS-Diff reduces SLO violations by over 5X, cuts latency by up to 2X, and increases throughput by 1.68X compared to the Stable Diffusion baseline.