Federated Learning for Decentralized Language Model Training across Global Data Sources

Federated learning has emerged as a powerful paradigm for training language models across decentralized data sources, enabling collaborative model development without the need to centralize sensitive data. This approach is particularly relevant for global-scale applications where data privacy, legal restrictions, and heterogeneous data distributions pose significant challenges. Existing methods for language model training often rely on centralized data aggregation, which can lead to privacy breaches, high communication costs, and limited adaptability to diverse local datasets. To address these challenges, we propose a Federated Averaging with Differential Privacy (FedAvg-DP) framework. In this framework, individual clients train local language models on their private datasets and share only differentially private model updates with a central aggregator. The aggregator performs weighted averaging to update the global model while ensuring that no sensitive information is exposed. This approach mitigates privacy risks, accommodates heterogeneous data distributions, and reduces network communication overhead. The proposed method is applied to cross-lingual healthcare chatbot development, allowing hospitals and clinics across multiple countries to collaboratively improve a global medical language model without sharing patient data. Experimental results demonstrate that FedAvg-DP achieves high accuracy, robust generalization across languages, and strong privacy preservation, confirming its effectiveness for decentralized language model training.