American Sign Language Recognition Using Leap Motion Controller with Machine Learning Approach

• Main Authors: Teak-Wei Chong, Boon-Giin Lee
• Format: Article
• Language: English
• Published: MDPI AG 2018-10-01
• Series: Sensors
• Subjects: human-computer interaction; machine learning; sign language recognition; Leap Motion Controller; support vector machine; deep neural network; multi-class classification; American Sign Language
• Online Access: http://www.mdpi.com/1424-8220/18/10/3554

Sign language is intentionally designed to allow deaf and dumb communities to convey messages and to connect with society. Unfortunately, learning and practicing sign language is not common among society; hence, this study developed a sign language recognition prototype using the Leap Motion Controller (LMC). Many existing studies have proposed methods for incomplete sign language recognition, whereas this study aimed for full American Sign Language (ASL) recognition, which consists of 26 letters and 10 digits. Most of the ASL letters are static (no movement), but certain ASL letters are dynamic (they require certain movements). Thus, this study also aimed to extract features from finger and hand motions to differentiate between the static and dynamic gestures. The experimental results revealed that the sign language recognition rates for the 26 letters using a support vector machine (SVM) and a deep neural network (DNN) are 80.30% and 93.81%, respectively. Meanwhile, the recognition rates for a combination of 26 letters and 10 digits are slightly lower, approximately 72.79% for the SVM and 88.79% for the DNN. As a result, the sign language recognition system has great potential for reducing the gap between deaf and dumb communities and others. The proposed prototype could also serve as an interpreter for the deaf and dumb in everyday life in service sectors, such as at the bank or post office.