Answers are grounded in this paper and cite their sources.
Multi-task distillation improves performance in low-resource languages, enhancing retrieval accuracy in legal applications.
Embedding models are a critical component of Retrieval-Augmented Generation (RAG) systems, but creating efficient, high-performance models for low-resource languages like Azerbaijani presents a significant challenge. To address this, we propose a multi-task distillation framework that improves a target low-resource language by distilling knowledge from English, while crucially preserving the model's original English language performance. Our approach combines cross-lingual knowledge distillation with a bilingual training objective, controlled by a single hyperparameter that balances performance across languages. We release a large-scale English-Azerbaijani parallel corpus of 300,000 Wikipedia articles to facilitate further research. Our experimental results demonstrate that models trained with our framework achieve state-of-the-art performance on Azerbaijani benchmarks: our BGE M3 model improves Azerbaijani STS from 79.3 to 85.0 (+5.7 points) while retaining 98.6% of its original English MTEB performance. In a real-world RAG application for Azerbaijani legal document retrieval, our best model achieves 89% top-1 retrieval accuracy and a 94% "Excellent" rating from domain experts, significantly outperforming baseline models. This framework provides a generalizable blueprint for enhancing embedding models for other low-resource languages, particularly within the Turkic language family.
No takes yet. Share an insight, caveat, or question.
Aghalarov et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: