Federated learning (FL) enables deployment of smart applications, where the intelligent model is trained with distributed data to achieve high accuracy and generalization capabilities. However, resource-constrained terminal devices are unable to train a global model at scale, leading to device dropouts and reduced data utilization. To achieve effective FL among heterogeneous models, we propose the edge learning based trusted heterogeneous federated learning scheme (ELTFL), which allows clients with different heterogeneous models to participate in FL through the application of knowledge distillation (KD). Specifically, ELTFL utilizes integrated distillation to build a heterogeneous multi-model training architecture for FL, which downloads global models to edge servers while preserving heterogeneous models of terminal devices. The edge server utilizes a small set of datasets along with probabilistic outputs from terminal devices to train the global model, and performs global model aggregation in the cloud. In addition, to safeguard against gradient leakage attacks, we introduce a lightweight iterative masking technique between edge servers and terminal devices to ensure the privacy and reliability of intermediate information. Extensive experiments on the CIFAR10 and CIFAR100 datasets demonstrate that ELTFL performs well in terms of accuracy, robustness to data heterogeneity, communication overhead, and privacy protection.
Lai et al. (Mon,) studied this question.