PMID- 31281206
OWN - NLM
STAT- PubMed-not-MEDLINE
LR  - 20231012
IS  - 0922-6567 (Print)
IS  - 1573-0573 (Electronic)
IS  - 0922-6567 (Linking)
VI  - 33
IP  - 1
DP  - 2019
TI  - An error analysis for image-based multi-modal neural machine translation.
PG  - 155-177
LID - 10.1007/s10590-019-09226-9 [doi]
AB  - In this article, we conduct an extensive quantitative error analysis of different 
      multi-modal neural machine translation (MNMT) models which integrate visual 
      features into different parts of both the encoder and the decoder. We investigate 
      the scenario where models are trained on an in-domain training data set of 
      parallel sentence pairs with images. We analyse two different types of MNMT 
      models, that use global and local image features: the latter encode an image 
      globally, i.e. there is one feature vector representing an entire image, whereas 
      the former encode spatial information, i.e. there are multiple feature vectors, 
      each encoding different portions of the image. We conduct an error analysis of 
      translations generated by different MNMT models as well as text-only baselines, 
      where we study how multi-modal models compare when translating both visual and 
      non-visual terms. In general, we find that the additional multi-modal signals 
      consistently improve translations, even more so when using simpler MNMT models 
      that use global visual features. We also find that not only translations of terms 
      with a strong visual connotation are improved, but almost all kinds of errors 
      decreased when using multi-modal models.
FAU - Calixto, Iacer
AU  - Calixto I
AUID- ORCID: 0000-0001-6244-7906
AD  - 1University of Amsterdam, ILLC, Science Park, Amsterdam, Netherlands. ISNI: 
      0000000084992262. GRID: grid.7177.6
FAU - Liu, Qun
AU  - Liu Q
AD  - Huawei Noah's Ark Lab, Hong Kong, Hong Kong.
LA  - eng
GR  - FC001000/ARC_/Arthritis Research UK/United Kingdom
PT  - Journal Article
DEP - 20190408
PL  - Netherlands
TA  - Mach Transl
JT  - Machine translation : MT
JID - 9884454
PMC - PMC6579783
OTO - NOTNLM
OT  - Error analysis
OT  - Machine translation
OT  - Multi-modal machine translation
OT  - Multi-modal neural machine translation
OT  - Neural machine translation
EDAT- 2019/07/10 06:00
MHDA- 2019/07/10 06:01
PMCR- 2019/04/08
CRDT- 2019/07/09 06:00
PHST- 2018/07/15 00:00 [received]
PHST- 2019/01/29 00:00 [accepted]
PHST- 2019/07/09 06:00 [entrez]
PHST- 2019/07/10 06:00 [pubmed]
PHST- 2019/07/10 06:01 [medline]
PHST- 2019/04/08 00:00 [pmc-release]
AID - 9226 [pii]
AID - 10.1007/s10590-019-09226-9 [doi]
PST - ppublish
SO  - Mach Transl. 2019;33(1):155-177. doi: 10.1007/s10590-019-09226-9. Epub 2019 Apr 
      8.