Composable Instructions and Prospection Guided Visuomotor Control for Robotic Manipulation

• Main Authors: Quanquan Shao, Jie Hu, Weiming Wang, Yi Fang, Mingshuo Han, Jin Qi, Jin Ma
• Format: Article
• Language: English
• Published: Atlantis Press 2019-10-01
• Series: International Journal of Computational Intelligence Systems
• Subjects: Composable instructions; Motion generation; Prospection; Imitation learning; Visuomotor control; Robotic manipulation
• Online Access: https://www.atlantis-press.com/article/125921091/view

Deep neural network-based end-to-end visuomotor control for robotic manipulation is becoming a hot issue of robotics field recently. One-hot vector is often used for multi-task situation in this framework. However, it is inflexible using one-hot vector to describe multiple tasks and transmit intentions of humans. This paper proposes a framework by combining composable instructions with visuomotor control for multi-task problems. The framework mainly consists of two modules: variational autoencoder (VAE) networks and long short-term memory (LSTM) networks. Perception information of the environment is encoded by VAE into a small latent space. The embedded perception information and composable instructions are combined by the LSTM module to guide robotic motion based on different intentions. Prospection is also used to learn the purposes of instructions, which means not only predicting the next action but also predicting a sequence of future actions at the same time. To evaluate this framework, a series of experiments are conducted in pick-and-place application scenarios. For new tasks, the framework could obtain a success rate of 91.2%, which means it has a good generalization ability.