With the proliferation of IoT technologies and the development of edge computing, achieving efficient deep learning inference on resource-constrained edge devices has become a critical challenge. This paper proposes a model compression framework for IoT edge devices based on knowledge distillation (PKD), aimed at effectively compressing models while enhancing their robustness and privacy protection on edge devices. Specifically, the proposed PKD framework introduces three key technical components: (1) a partitioned knowledge distillation strategy that enables efficient deployment of teacher models in distributed edge environments, (2) a Gradient-Weighted Feature Attention Loss (GW-FAL) mechanism to improve feature-level knowledge transfer, and (3) a Targeted Fast Gradient Perturbation (TFGP) method to enhance model robustness against adversarial disturbances. By partitioning the teacher model deployment, applying GW-FAL to improve knowledge transfer, and introducing adversarial perturbations to enhance model robustness, PKD achieves a high compression rate without significantly sacrificing accuracy. Through extensive experiments, the PKD framework demonstrates superior noise robustness and defense against adversarial attacks while improving model compression efficiency. The experimental results show that the framework provides an ideal balance between accuracy and speed on resource-constrained devices, making it suitable for practical applications such as IoT.
Lin et al. (Thu,) studied this question.