Gated transformer with shallow decoder for machine translation

Large language models (LLMs) have substantially advanced multilingual translation, yet their computational cost limits deployment in resource-constrained scenarios and latency-critical applications. Traditional Transformer-based neural machine translation (NMT) systems remain valuable in these settings. In research on improving the efficiency of the Transformer, linearization techniques that reduce the time complexity of the Transformer to O(N) only show advantages for very long texts. To address this gap, we propose a lightweight architecture, Gated Transformer with Shallow Decoder (GTSD), designed specifically for low-cost and short-text translation. The proposed method employs a gating mechanism to fuse attention and the feed-forward network (FFN), optimizes the redundant cross-attention resulting from the transformation from multi-head to single-head, and adopts a deep encoder-shallow decoder architecture. Furthermore, the proposed method supports simple cost reduction with minimal loss in accuracy. Finally, a series of experiments conducted on WMT14 en-de and WMT14 en-fr datasets demonstrate the proposed approach attains a strong efficiency–performance trade-off.