Neural Machine Translation – NLP White Paper Part 2

2025.8.21
Laboro.AI Inc.　Lead ML Researcher　Zhao Xinyi

日本語版（Japanese version）はこちら

Here is NLP White Paper – Part1: Overview

Introduction

Since 2017, the Transformer architecture has dominated Neural Machine Translation (NMT), powering influential models like XLM, mBART, and mT5. Its encoder-decoder design effectively leverages parallel corpora. More recently, decoder-only large language models have emerged, enabling NMT to benefit from non-parallel data and carefully designed prompts. Despite this progress, low-resource and distant language pairs remain challenging. Approaches such as meta-learning, few-shot, and zero-shot translation aim to bridge these gaps via cross-lingual knowledge transfer. As the field evolves, new evaluation practices like COMET and BUFFET are complementing traditional metrics like BLEU, offering more reliable evaluations aligned with human judgments.

Contents

・Introduction to Neural Machine Translation
・Core Breakthroughs
・Future Directions & Challenges
　・Low-Resource Neural Machine Translation
　・Best Practices for LLM-Based Translation

Introduction to Neural Machine Translation

The field of Natural Language Processing has seen a major transformation in recent years, with Neural Machine Translation (NMT) playing a leading role. By leveraging deep learning, NMT has significantly surpassed traditional Statistical Machine Translation (SMT) in both translation accuracy and fluency. Unlike SMT, which relied on statistical phrase matching and probabilistic models, NMT treats translation as a single, end-to-end learning problem, enabling it to better capture the meaning of full sentences and produce more fluent results. A typical NMT system is built on sequence-to-sequence (seq2seq) models, mapping an input sequence in the source language directly to an output sequence in the target language, setting a foundation for more flexible and powerful translation systems.

Core Breakthroughs

Early NMT systems were built on Recurrent Neural Networks (RNNs), which struggled with longer sentences and complex dependencies. A major turning point came with the introduction of the Transformer model (2017), which replaced recurrence with self-attention, enabling faster, more accurate translation with more effective handling of long sequences. Today, Transformers form the backbone of nearly all modern translation systems and have paved the way for the broader development of Large Language Models (LLMs).

Training Transformer-based NMT models typically follows an encoder-decoder structure and depends on large parallel datasets, either focused on a specific language pair or spanning many languages at once. Notable models from this era such as XLM (2019), mBART (2020), and mT5 (2021) pushed multilingual translation forward by training Transformer models on massive multilingual datasets. These developments have made high-quality machine translation more accessible across a wider range of languages.

Another breakthrough has been in the evaluation of translation quality. Traditional metrics like BLEU (2002) often miss nuances important to human readers. Newer frameworks like COMET (2020) use pre-trained cross-lingual models to predict translation quality, leading to higher correlation to human judgment. This innovation points to a larger trend: integrating smarter evaluation tools directly into translation workflows, ensuring not just grammatical correctness but also greater contextual and cultural relevance, which are critical factors for business applications across global markets.

Future Directions & Challenges

Low-Resource Neural Machine Translation

One of the biggest challenges in machine translation today is handling languages with limited available training data, often referred to as low-resource translation. While major languages benefit from decades of data, many global and regional languages are left behind. Recent breakthroughs show that it’s possible to improve performance on these low-resource languages by leveraging cross-lingual knowledge transfer, where models trained on high-resource languages help improve translation quality in low-resource ones.

For example, the concept of meta-learning, or learning-to-learn, was explored for NMT by Gu et al., 2018, aiming at fast adaptation on low-resource languages. Another approach proposed by Lin et al., 2020 brings semantically related phrases across languages closer together in the text representation space. Additionally, studies like Aharoni et al., 2019 and Xue et al., 2021 have shown that training a single model on diverse languages simultaneously can naturally benefit low-resource languages. These advances offer a promising path toward more inclusive, equitable translation systems for underrepresented communities.

Best Practices for LLM-Based Translation

With the rise of LLMs, a new paradigm for translation is emerging. Unlike traditional encoder-decoder models trained on parallel data, LLM-based translation typically depends on decoder-only models trained on large-scale, non-parallel datasets. Research by Vu et al., 2022 shows that even unlabeled multilingual corpora can significantly improve zero-shot translation when incorporated into LLM training.

Recent studies further show that LLMs can produce high-quality translations, but how you prompt them matters a lot. For instance, Peng et al., 2023 revisits key aspects such as temperature, task specification, and domain adaptation, exploring how different prompting strategies affect ChatGPT’s translation performance. Other work such as Agrawal et al., 2023 and  Vilar et al., 2023 emphasizes the importance of choosing good examples in few-shot settings, and reveals low-quality examples can significantly degrade translation quality. In response, benchmarks like BUFFET were designed to evaluate how well LLMs handle few-shot tasks across languages.

While LLMs offer exciting new possibilities for translation, especially in multilingual and low-resource scenarios, they also come with trade-offs. Successful translation often depends on careful prompt engineering, data selection, and a clear understanding of where LLMs shine versus where traditional NMT systems may still be more reliable.

Here is NLP White Paper – Part1: Overview

Author

Laboro.AI Inc.　Lead ML Researcher　Zhao Xinyi

Xinyi Zhao is a lead researcher at Laboro.AI Inc. Her research focuses on natural language processing, machine learning, and kowledge graphs. She has contributed multiple open-source datasets and models, and her recent work explores real-world applications of large language models. She’s passionate about bridging academic research with practical use cases.