Facial Emotion Recognition from an Unmanned Flying Social Robot for Home Care of Dependent People

• Main Authors: Anselmo Martínez, Lidia M. Belmonte, Arturo S. García, Antonio Fernández-Caballero, Rafael Morales
• Format: Article
• Language: English
• Published: MDPI AG 2021-04-01
• Series: Electronics
• Subjects: flying social robot; autonomous unmanned aerial vehicle (UAV); emotion recognition; convolution neural network (CNN); virtual reality (VR); unity
• Online Access: https://www.mdpi.com/2079-9292/10/7/868

This work is part of an ongoing research project to develop an unmanned flying social robot to monitor dependants at home in order to detect the person’s state and bring the necessary assistance. In this sense, this paper focuses on the description of a virtual reality (VR) simulation platform for the monitoring process of an avatar in a virtual home by a rotatory-wing autonomous unmanned aerial vehicle (UAV). This platform is based on a distributed architecture composed of three modules communicated through the message queue telemetry transport (MQTT) protocol: the UAV Simulator implemented in MATLAB/Simulink, the VR Visualiser developed in Unity, and the new emotion recognition (ER) system developed in Python. Using a face detection algorithm and a convolutional neural network (CNN), the ER System is able to detect the person’s face in the image captured by the UAV’s on-board camera and classify the emotion among seven possible ones (surprise; fear; happiness; sadness; disgust; anger; or neutral expression). The experimental results demonstrate the correct integration of this new computer vision module within the VR platform, as well as the good performance of the designed CNN, with around 85% in the F1-score, a mean of the precision and recall of the model. The developed emotion detection system can be used in the future implementation of the assistance UAV that monitors dependent people in a real environment, since the methodology used is valid for images of real people.