Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to...

Full description

Bibliographic Details
Main Authors: Schuster, Tal (Author), Ram, Ori (Author), Barzilay, Regina (Author), Globerson, Amir (Author)
Other Authors: Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory (Contributor), Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science (Contributor)
Format: Article
Language:English
Published: Association for Computational Linguistics, 2020-12-02T20:59:53Z.
Subjects:
Online Access:Get fulltext
LEADER 01988 am a22002173u 4500
001 128715
042 |a dc 
100 1 0 |a Schuster, Tal  |e author 
100 1 0 |a Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory  |e contributor 
100 1 0 |a Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science  |e contributor 
700 1 0 |a Ram, Ori  |e author 
700 1 0 |a Barzilay, Regina  |e author 
700 1 0 |a Globerson, Amir  |e author 
245 0 0 |a Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing 
260 |b Association for Computational Linguistics,   |c 2020-12-02T20:59:53Z. 
856 |z Get fulltext  |u https://hdl.handle.net/1721.1/128715 
520 |a We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average. 
520 |a Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA) (Contract FA8650-17-C- 9116) 
546 |a en 
655 7 |a Article 
773 |t 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies