Leveraging the Visual Capabilities of Transformers in Multimodal Machine Translation

Machine translation (MT) has come a long way since Deep Neural Networks (DNNs) arrived. The introduction of Transformer architecture, with its flexible data handling, opened the door to a new field: Multimodal Machine Translation (MMT). MMT aims to combine text with other information, like images, to improve translation accuracy. While MMT is a rapidly growing field, there are still challenges. One is the lack of data that combines different modalities with translations. Another is how to represent different data types effectively and then combine them in a way that captures the overall meaning. This thesis proposes a new architecture using three transformers: one for text, one for a general image representation, and one for detecting objects in the image. The goal is to see if using both general and specific image features improves translation quality. Additionally, this research focuses on a lightweight architecture compared to the current trend of using increasingly complex models. Experiments were conducted using two Transformer sizes ("Tiny" and "Small") for translating English to German, French, and Czech. The results show that the proposed approach works well for German and French, but for Czech, only the general image representation led to improvements so far.