• continual learning
• neural machine translation
• feature transferability
• continual neural machine translation
• lifelong learning

The aim of this work is to investigate the impact of pre-training on neural machine translation. In the context of continual learning, we assess how the incremental addition of new languages in the pre-training phase improves or degrades the performance of translation models on both past and current experience. We also assessed how much the features learned by the models changed during both the incremental pre-training phase and the fine-tuning one.

The languages considered were the most commonly used, namely English, French, German and Spanish. In the first experience, a model was pre-trained on one language pair. In each subsequent experience, the model from the previous experience is trained on the newly introduced language.
At the end of this process, we fine-tune all the pre-trained models on the aforementioned languages, using an English-centric approach, i.e. translating from English to another language and vice versa.

In this way, we can assess the extent to which the incremental addition of languages causes the representations learned by the models during past experience to be forgotten. Finally, we apply the random replay technique to the pre-training phase and evaluate its impact on the downstream task.