Speech Translation into Pakistan Sign Language

• Main Authors: Haseeb, Ahmed Abdul, Ilyas, Asim
• Format: Others
• Language: English
• Published: Blekinge Tekniska Högskola, Sektionen för datavetenskap och kommunikation 2012
• Subjects: Sign language; Pakistan sign language; Speech to sign language converter; speech and Pakistan sign language; Sign language interpreting machine; user centric; prototype validation; Computer Sciences; Datavetenskap (datalogi)
• Online Access: http://urn.kb.se/resolve?urn=urn:nbn:se:bth-5095

ABSTRACT Context: Communication is a primary human need and language is the medium for this. Most people have the ability to listen and speak and they use different languages like Swedish, Urdu and English etc. to communicate. Hearing impaired people use signs to communicate. Pakistan Sign Language (PSL) is the preferred language of the deaf in Pakistan. Currently, human PSL interpreters are required to facilitate communication between the deaf and hearing; they are not always available, which means that communication among the deaf and other people may be impaired or nonexistent. In this situation, a system with voice recognition as an input and PSL as an output will be highly helpful. Objectives: As part of this thesis, we explore challenges faced by deaf people in everyday life while interacting with unimpaired. We investigate state of art work done in this area. This study explores speech recognition and Machine translation techniques to devise a generic and automated system that converts English speech to PSL. A prototype of the proposed solution is developed and validated. Methods: Three step investigation is done as part of thesis work. First, to understand problem itself, interviews were conducted with the domain experts. Secondly, from literature review, it is investigated whether any similar or related work has already been done, state of the art technologies like Machine translation, speech recognition engines and Natural language processing etc. have been analyzed. Thirdly, prototype is developed whose validation data is obtained from domain experts and is validated by ourselves as well as from domain experts. Results: It is found that there is a big communication gap between deaf and unimpaired in Pakistan. This is mainly due to the lack of an automated system that can convert Audio speech to PSL and vice versa. After investigating state of the art work including solutions in other countries specific to their languages, it is found that no system exists that is generic and automated. We found that there is already work started for PSL to English Speech conversion but not the other way around. As part of this thesis, we discovered that a generic and automated system can be devised using speech recognition and Machine translation techniques. Conclusion: Deaf people in Pakistan lack a lot of opportunities mainly due to communication gap between deaf and unimpaired. We establish that there should be a generic and automated system that can convert English speech to PSL and vice versa. As part of this, we worked for such a system that can convert English speech to PSL. Moreover, Speech recognition, Machine translation and Natural language processing techniques can be core ingredients for such a generic and automated system. Using user centric approach, the prototype of the system is validated iteratively from domain experts. === This research has investigated a computer based solution to facilitate communication among deaf people and unimpaired. Investigation was performed using literature review and visits to institutes to gain a deeper knowledge about sign language and specifically how is it used in Pakistan context. Secondly, challenges faced by deaf people to interact with unimpaired are analyzed by interviews with domain experts (instructors of deaf institutes) and by directly observing deaf in everyday life situations. We conclude that deaf people rely on sign language for communication with unimpaired people. Deaf people in Pakistan use PSL for communication, English is taught as secondary language all over Pakistan in all educational institutes, deaf people are taught by instructors that not only need to know the domain expertise of the area that they are teaching like Math, History and Science etc. but they also need to know PSL very well in order to teach the deaf. It becomes very difficult for deaf institutes to get instructors that know both. Whenever deaf people need to communicate with unimpaired people in any situation, they either need to hire a translator or request the unimpaired people to write everything for them. Translators are very difficult to get all the time and they are very expensive as well. Moreover, using writing by unimpaired becomes very slow process and not all unimpaired people want to do this. We observed this phenomena ourselves as instructors of the institutes provided us the opportunity to work with deaf people to understand their feelings and challenges in everyday life. In this way, we used to go with deaf people in shopping malls, banks, post offices etc. and with their permission, we observed their interaction. We have concluded that sometimes their interaction with normal people becomes very slow and embarrassing. Based on above findings, we concluded that there is definitely a need for an automated system that can facilitate communication between deaf and unimpaired people. These factors lead to the subsequent objective of this research. The main objective of this thesis is to identify a generic and an automated system without any human intervention that converts English speech into PSL as a solution to bridge the communication gap between deaf and unimpaired. It is identified that existing work done related to this problem area doesn’t fulfill our objective. Current solutions are either very specific to a domain, e.g. post office or need human intervention i.e. not automatic. It is identified that none of the existing systems can be extended towards our desired solution. We explored state of the art techniques like Machine translation, Speech recognition and NLP. We have utilized these in our proposed solution. Prototype of the proposed solution is developed whose functional and non functional validation is performed. Since none of existing work exactly matches to our problem statement, therefore, we have not compared the validation of our prototype to any existing system. We have validated prototype with respect to our problem domain. Moreover, this is validated iteratively from the domain experts, i.e. experts of PSL and the English to PSL human translators. We found this user centric approach very useful to help better understand the problem at the ground level, keeping our work user focused and then realization of user satisfaction level throughout the process. This work has opened a new world of opportunities where deaf can communicate with others who do not have PSL knowledge. Having this system, if it is further developed from a prototype to a functioning system; deaf institutes will have wider scope of choosing best instructors for a given domain that may not have PSL expertise. Deaf people will have more opportunities to interact with other members of the society at every level as communication is the basic pillar for this. The automatic speech to sign language is an attractive prospect; the impending applications are exhilarating and worthwhile. In the field of Human Computer Interface (HCI) we hope that our thesis will be an important addition to the ongoing research. === Ahmed Abdul Haseeb & Asim ilyas, Contact no. 00923215126749 House No. 310, Street No. 4 Rawal town Islamabad, Pakistan Postal Code 44000