Most existing neural machine translation (NMT) models extract the source representations from lower layers to higher layers without external guidance signals, and the encoder output is used as the final source language representation. In human understanding, the iteration between bottom-up and top-down processes plays a crucial role in comprehension. That is to say, human understanding is full of feedback signals and repeated processing. To obtain a better source language representation, we integrate feedback or recurrent signals in the traditional NMT encoder. This iterative refinement mechanism aligns with the principles of data intelligence, where richer insights are derived through dynamic, context-aware processing of information rather than static, one-pass analysis. To this end, our work presents an approach that is both straightforward and effective. It focuses on learning iterative feature representations from input sentences by means of a recurrent feature encoding (RFE) model. Here, the iterative representation is repeatedly refined based on the feedback context from the previous iteration’s output. Through the proposed RFE, we can progressively obtain a better source representation by the effective interaction between bottom-up and top-down information—enabling the model to exhibit more intelligent behavior in capturing complex linguistic patterns, a key goal of data intelligence. We conduct experiments on the commonly utilized, extensive workshop on statistical machine translation (WMT) 2016 English-to-German (En-De) and WMT 2014 English-to-French (En-Fr) translation tasks. The experimental results show that our proposed iterative feature refinement (IFR) by an RFE significantly outperforms the strong baseline. Moreover, compared with recent deep NMT models and Transformer variants, our approach with Transformer big setting has achieved comparable performance on the two large-scale tasks. Another valuable finding is that we propose a simple Transformer encoder to repeatedly refine the source sentence representations and achieve comparable performance with the state-of-the-art deep NMT models. This demonstrates that enhancing the depth of data interpretation through iterative refinement, rather than merely increasing architectural depth, is a promising direction for advancing data intelligence in natural language processing systems.
Zhang et al. (Fri,) studied this question.