Dynamic Model and Node Selection for Collaborative Inference of Large/Small Models in Vehicular Networks

Collaborative inference between large cloud-hosted models and small edge-deployed models offers a promising solution for balancing the accuracy and efficiency of ML-based applications in vehicular networks. Selecting the appropriate models and their hosting nodes for performing various inference tasks plays a crucial role in collaborative inference in vehicular networks. However, existing solutions, primarily based on deep reinforcement learning (DRL), suffer critical limitations, including delayed and suboptimal decisions on model and node selection in dynamic environments. To address these challenges, we propose a dynamic model and node selection strategy for a collaborative inference framework, grounded in active inference theory. Our strategy dynamically aligns task requirements with model capabilities and node capacities by considering factors such as vehicular mobility, latency constraints, task complexity, and model accuracy. Additionally, when significant drops in inference accuracy are detected, we fine-tune and update the models deployed on both the edge and cloud, ensuring reliable and up-to-date inference. By leveraging active inference to minimize free energy through Bayesian belief updates, our framework reduces average latency by 23.2%, lowers task failure rates by 67%, and achieves superior load balancing compared to existing methods. It also demonstrates robust dynamic performance with a 5.1% failure rate under 200% traffic surges, and its hybrid update strategy maintains 85.4% accuracy after 72 hours, effectively addressing the complex and dynamic conditions of vehicular networks.