Validating open-source machine translation for quantitative text analysis

Abstract Machine translation (MT) is an essential tool in many multilingual computational text analysis applications. However, relying on commercial services like Google Translate or DeepL limits reproducibility and can be expensive. This paper assesses the viability of a reproducible, transparent, and affordable alternative: open-source MT models. We ask whether using open-source MT models instead of commercial services substantially changes the measurements obtained from multilingual corpora by extending an influential study by de Vries et al. and contributing an original study focusing on Transformer-based supervised text classification. Our findings reveal negligible differences in results between the two MT approaches, suggesting that open-source MT models are highly valuable tools for multilingual text analysis.