Risk-Aware Model-Based Control

• Main Authors: Chen Yu, Andre Rosendo
• Format: Article
• Language: English
• Published: Frontiers Media S.A. 2021-03-01
• Series: Frontiers in Robotics and AI
• Subjects: machine learning; reinforcement learning; dynamics model; risk awareness; conditional value at risk; data efficiency
• Online Access: https://www.frontiersin.org/articles/10.3389/frobt.2021.617839/full

Model-Based Reinforcement Learning (MBRL) algorithms have been shown to have an advantage on data-efficiency, but often overshadowed by state-of-the-art model-free methods in performance, especially when facing high-dimensional and complex problems. In this work, a novel MBRL method is proposed, called Risk-Aware Model-Based Control (RAMCO). It combines uncertainty-aware deep dynamics models and the risk assessment technique Conditional Value at Risk (CVaR). This mechanism is appropriate for real-world application since it takes epistemic risk into consideration. In addition, we use a model-free solver to produce warm-up training data, and this setting improves the performance in low-dimensional environments and covers the shortage of MBRL’s nature in the high-dimensional scenarios. In comparison with other state-of-the-art reinforcement learning algorithms, we show that it produces superior results on a walking robot model. We also evaluate the method with an Eidos environment, which is a novel experimental method with multi-dimensional randomly initialized deep neural networks to measure the performance of any reinforcement learning algorithm, and the advantages of RAMCO are highlighted.