Attention Mechanisms and Their Applications to Complex Systems

• Main Authors: Adrián Hernández, José M. Amigó
• Format: Article
• Language: English
• Published: MDPI AG 2021-02-01
• Series: Entropy
• Subjects: attention; deep learning; complex and dynamical systems; self-attention; neural networks; sequential reasoning
• Online Access: https://www.mdpi.com/1099-4300/23/3/283

Deep learning models and graphics processing units have completely transformed the field of machine learning. Recurrent neural networks and long short-term memories have been successfully used to model and predict complex systems. However, these classic models do not perform sequential reasoning, a process that guides a task based on perception and memory. In recent years, attention mechanisms have emerged as a promising solution to these problems. In this review, we describe the key aspects of attention mechanisms and some relevant attention techniques and point out why they are a remarkable advance in machine learning. Then, we illustrate some important applications of these techniques in the modeling of complex systems.