Language modeling and bidirectional coders representations: an overview of key technologies

• Main Author: D. I. Kachkou
• Format: Article
• Language: Russian
• Published: The United Institute of Informatics Problems of the National Academy of Sciences of Belarus 2021-01-01
• Series: Informatika
• Subjects: informatics; information technology; language models; natural language processing; attention mechanism; transformer architecture; model bert
• Online Access: https://inf.grid.by/jour/article/view/1080

The article is an essay on the development of technologies for natural language processing, which formed the basis of BERT (Bidirectional Encoder Representations from Transformers), a language model from Google, showing high results on the whole class of problems associated with the understanding of natural language. Two key ideas implemented in BERT are knowledge transfer and attention mechanism. The model is designed to solve two problems on a large unlabeled data set and can reuse the identified language patterns for effective learning for a specific text processing problem. Architecture Transformer is based on the attention mechanism, i.e. it involves evaluation of relationships between input data tokens. In addition, the article notes strengths and weaknesses of BERT and the directions for further model improvement.