Skelesense : Arabic sign language recognizer, communicator and tutor for the deaf and mute

• Main Author: Khadregi, Ahmed Ibrahim
• Published: University of Salford 2011
• Subjects: 371.9
• Online Access: http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.543504

The deaf and mute society is a closed society that faces many obstacles in communication with the outside world. Through the use of the sign language the deaf and mute people express themselves to each other and to those who know their signs. There are many sign languages worldwide. British Sign Language (BSL) and American Sign Language (ASL) are amongst the most highly used and attention receiving sign languages. One of the less evolving sign languages is the Arabic Sign Language (ArSL) in general and its Egyptian Sign Language (ESL) dialect in specific. There are very few resources and fewer researches that were conducted to promote the ArSL or the ESL. So, the aim of this work is to develop and evaluate a computer-based system, SkeleSense, to teach the deaf and mute their sign language and enhance their communication skills with the surrounding world. This thesis describes the design and implementation of this system which is used for the recognition, teaching and communication of ESL. The approach taken is to develop a low-cost sensing glove which provides movement signals which are input to a computer system and used to control a 3D module of the user's hand. A graphical interface provides a user friendly environment allowing the user to choose various modes of operation for the system, including training of a user's hand-movements and comparison with expert's pre-stored sign movements for different words. SkeleSense allows the deaf to communicate with others through the production of sign to speech translator recognizing input signs and converting them into sound. Four local ESL experts helped to produce a 65 evaluation sign pool. Ten deaf users tested the system by comparing their signs against the experts' stored references. Each user was allowed 5 trials for each of the 65 signs. SkeleSense users achieved an overall average correctness of 4.69 out of 5 (93.8%). The highest average correctness was 4.92 out of 5 (98.4%) and the lowest was 4.09 (82%). Based on users' scores, 18.5% of the signs were difficult to perform because of certain joints bending angles. Those signs specifically showed enhanced correctness measures with the use of SkeleSense. Questionnaires were conducted showing that users including the one who achieved the least correctness recommended the system's use and performance over the traditional human based sign language systems for tutoring ESL. This indicates the significant role of SkeleSense in sign language tutoring.