Real-Time Human Detection and Gesture Recognition for On-Board UAV Rescue

• Main Authors: Chang Liu, Tamás Szirányi
• Format: Article
• Language: English
• Published: MDPI AG 2021-03-01
• Series: Sensors
• Subjects: unmanned aerial vehicles (UAVs); search and rescue (SAR); UAV human communication; body gesture recognition; hand gesture recognition; neural networks
• Online Access: https://www.mdpi.com/1424-8220/21/6/2180

Unmanned aerial vehicles (UAVs) play an important role in numerous technical and scientific fields, especially in wilderness rescue. This paper carries out work on real-time UAV human detection and recognition of body and hand rescue gestures. We use body-featuring solutions to establish biometric communications, like yolo3-tiny for human detection. When the presence of a person is detected, the system will enter the gesture recognition phase, where the user and the drone can communicate briefly and effectively, avoiding the drawbacks of speech communication. A data-set of ten body rescue gestures (i.e., Kick, Punch, Squat, Stand, Attention, Cancel, Walk, Sit, Direction, and PhoneCall) has been created by a UAV on-board camera. The two most important gestures are the novel dynamic Attention and Cancel which represent the set and reset functions respectively. When the rescue gesture of the human body is recognized as Attention, the drone will gradually approach the user with a larger resolution for hand gesture recognition. The system achieves 99.80% accuracy on testing data in body gesture data-set and 94.71% accuracy on testing data in hand gesture data-set by using the deep learning method. Experiments conducted on real-time UAV cameras confirm our solution can achieve our expected UAV rescue purpose.