What makes multilingual BERT multilingual?

10/20/2020

∙

Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability.

READ FULL TEXT

What makes multilingual BERT multilingual?

A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT

Measuring Cross-Lingual Transferability of Multilingual Transformers on Sentence Classification

Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER

Multilingual Central Repository: a Cross-lingual Framework for Developing Wordnets

Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings

Probing Multilingual BERT for Genetic and Typological Signals

Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval

What makes multilingual BERT multilingual?

Related Research

A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT

Measuring Cross-Lingual Transferability of Multilingual Transformers on Sentence Classification

Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER

Multilingual Central Repository: a Cross-lingual Framework for Developing Wordnets

Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings

Probing Multilingual BERT for Genetic and Typological Signals

Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval