Learning Motor Skills: From Algorithms to Robot Experiments

• Main Author: Kober, Jens
• Format: Others
• Language: English; en
• Published: 2012
• Online Access: https://tuprints.ulb.tu-darmstadt.de/2992/1/DissertationKober.pdf; Kober, Jens <http://tuprints.ulb.tu-darmstadt.de/view/person/Kober=3AJens=3A=3A.html> (2012): Learning Motor Skills: From Algorithms to Robot Experiments.Darmstadt, Technische Universität, [Ph.D. Thesis]

Ever since the word "robot" was introduced to the English language by Karel Capek's play "Rossum's Universal Robots" in 1921, robots have been expected to become part of our daily lives. In recent years, robots such as autonomous vacuum cleaners, lawn mowers, and window cleaners, as well as a huge number of toys have been made commercially available. However, a lot of additional research is required to turn robots into versatile household helpers and companions. One of the many challenges is that robots are still very specialized and cannot easily adapt to changing environments and requirements. Since the 1960s, scientists attempt to provide robots with more autonomy, adaptability, and intelligence. Research in this field is still very active but has shifted focus from reasoning based methods towards statistical machine learning. Both navigation (i.e., moving in unknown or changing environments) and motor control (i.e., coordinating movements to perform skilled actions) are important sub-tasks. In this thesis, we will discuss approaches that allow robots to learn motor skills. We mainly consider tasks that need to take into account the dynamic behavior of the robot and its environment, where a kinematic movement plan is not sufficient. The presented tasks correspond to sports and games but the presented techniques will also be applicable to more mundane household tasks. Motor skills can often be represented by motor primitives. Such motor primitives encode elemental motions which can be generalized, sequenced, and combined to achieve more complex tasks. For example, a forehand and a backhand could be seen as two different motor primitives of playing table tennis. We show how motor primitives can be employed to learn motor skills on three different levels. First, we discuss how a single motor skill, represented by a motor primitive, can be learned using reinforcement learning. Second, we show how such learned motor primitives can be generalized to new situations. Finally, we present first steps towards using motor primitives in a hierarchical setting and how several motor primitives can be combined to achieve more complex tasks. To date, there have been a number of successful applications of learning motor primitives employing imitation learning. However, many interesting motor learning problems are high-dimensional reinforcement learning problems which are often beyond the reach of current reinforcement learning methods. We review research on reinforcement learning applied to robotics and point out key challenges and important strategies to render reinforcement learning tractable. Based on these insights, we introduce novel learning approaches both for single and generalized motor skills. For learning single motor skills, we study parametrized policy search methods and introduce a framework of reward-weighted imitation that allows us to derive both policy gradient methods and expectation-maximization (EM) inspired algorithms. We introduce a novel EM-inspired algorithm for policy learning that is particularly well-suited for motor primitives. We show that the proposed method out-performs several well-known parametrized policy search methods on an empirical benchmark both in simulation and on a real robot. We apply it in the context of motor learning and show that it can learn a complex ball-in-a-cup task on a real Barrett WAM. In order to avoid re-learning the complete movement, such single motor skills need to be generalized to new situations. In this thesis, we propose a method that learns to generalize parametrized motor plans, obtained by imitation or reinforcement learning, by adapting a small set of global parameters. We employ reinforcement learning to learn the required parameters to deal with the current situation. Therefore, we introduce an appropriate kernel-based reinforcement learning algorithm. To show its feasibility, we evaluate this algorithm on a toy example and compare it to several previous approaches. Subsequently, we apply the approach to two robot tasks, i.e., the generalization of throwing movements in darts and of hitting movements in table tennis on several different real robots, i.e., a Barrett WAM, the JST-ICORP/SARCOS CBi and a Kuka KR 6. We present first steps towards learning motor skills jointly with a higher level strategy and evaluate the approach with a target throwing task on a BioRob. Finally, we explore how several motor primitives, representing sub-tasks, can be combined and prioritized to achieve a more complex task. This learning framework is validated with a ball-bouncing task on a Barrett WAM. This thesis contributes to the state of the art in reinforcement learning applied to robotics both in terms of novel algorithms and applications. We have introduced the Policy learning by Weighting Exploration with the Returns algorithm for learning single motor skills and the Cost-regularized Kernel Regression to generalize motor skills to new situations. The applications explore highly dynamic tasks and exhibit a very efficient learning process. All proposed approaches have been extensively validated with benchmarks tasks, in simulation, and on real robots.