MTMLD-AWSR: A novel Multi-Teacher Multi-Level Distillation approach for Class Incremental Learning in Edge-Cloud systems

In practical Edge-Cloud applications, deep learning models often face challenges when learning from continuously generated new data and labels from IoT devices. As a result, the model tends to struggle to retain previous knowledge while absorbing new knowledge, especially information from a new label in a classification task. This problem is also known as Class Incremental Learning (CIL). Among the approaches for CIL, Knowledge Distillation (KD) has emerged as a promising method and has achieved notable success. However, this approach may not effectively capture long-term dependencies and diverse feature representations, leading to suboptimal performance. This paper introduces the Multi-Teacher Multi-level Distillation (MTMLD) into CIL, utilizing multiple teachers to improve knowledge transfer at different levels, including feature, logit, and attention. Additionally, to ensure efficiency and scalability, we propose two novel modules: (1) a Student Reusability Module, which allows the student model to serve as a teacher in subsequent tasks, reducing redundancy, and (2) a Teacher Management Module, which regulates the number of teachers to prevent unnecessary computational overhead. Extensive experiments comparing our method to other algorithms on multiple datasets demonstrate that the proposed framework significantly outperforms existing methods in terms of accuracy and reduction of forgetting, achieving improvements of up to 2.15% compared to state-of-the-art algorithms on the CIFAR-100 dataset.