3D Avatar Approach for Continuous Sign Movement Using Speech/Text

Sign language is a visual language for communication used by hearing-impaired people with the help of hand and finger movements. Indian Sign Language (ISL) is a well-developed and standard way of communication for hearing-impaired people living in India. However, other people who use spoken language...

Full description

Bibliographic Details
Main Authors: Debashis Das Chakladar, Pradeep Kumar, Shubham Mandal, Partha Pratim Roy, Masakazu Iwamura, Byung-Gyu Kim
Format: Article
Language:English
Published: MDPI AG 2021-04-01
Series:Applied Sciences
Subjects:
Online Access:https://www.mdpi.com/2076-3417/11/8/3439
id doaj-caefbe81c89f4e7687d161076427d6c1
record_format Article
spelling doaj-caefbe81c89f4e7687d161076427d6c12021-04-12T23:01:50ZengMDPI AGApplied Sciences2076-34172021-04-01113439343910.3390/app110834393D Avatar Approach for Continuous Sign Movement Using Speech/TextDebashis Das Chakladar0Pradeep Kumar1Shubham Mandal2Partha Pratim Roy3Masakazu Iwamura4Byung-Gyu Kim5Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Uttarakhand 247667, IndiaDepartment of Computer Science and Engineering, Indian Institute of Technology Roorkee, Uttarakhand 247667, IndiaDepartment of Computer Science and Engineering, Indian Institute of Technology Roorkee, Uttarakhand 247667, IndiaDepartment of Computer Science and Engineering, Indian Institute of Technology Roorkee, Uttarakhand 247667, IndiaDepartment of Computer Science and Intelligent Systems, Osaka Prefecture University, Osaka 599-8531, JapanDepartment of IT Engineering, Sookmyung Women’s University, Seoul 140-742, KoreaSign language is a visual language for communication used by hearing-impaired people with the help of hand and finger movements. Indian Sign Language (ISL) is a well-developed and standard way of communication for hearing-impaired people living in India. However, other people who use spoken language always face difficulty while communicating with a hearing-impaired person due to lack of sign language knowledge. In this study, we have developed a 3D avatar-based sign language learning system that converts the input speech/text into corresponding sign movements for ISL. The system consists of three modules. Initially, the input speech is converted into an English sentence. Then, that English sentence is converted into the corresponding ISL sentence using the Natural Language Processing (NLP) technique. Finally, the motion of the 3D avatar is defined based on the ISL sentence. The translation module achieves a 10.50 SER (Sign Error Rate) score.https://www.mdpi.com/2076-3417/11/8/3439Indian Sign Language (ISL)natural language processingavatarsign movementcontext-free grammar
collection DOAJ
language English
format Article
sources DOAJ
author Debashis Das Chakladar
Pradeep Kumar
Shubham Mandal
Partha Pratim Roy
Masakazu Iwamura
Byung-Gyu Kim
spellingShingle Debashis Das Chakladar
Pradeep Kumar
Shubham Mandal
Partha Pratim Roy
Masakazu Iwamura
Byung-Gyu Kim
3D Avatar Approach for Continuous Sign Movement Using Speech/Text
Applied Sciences
Indian Sign Language (ISL)
natural language processing
avatar
sign movement
context-free grammar
author_facet Debashis Das Chakladar
Pradeep Kumar
Shubham Mandal
Partha Pratim Roy
Masakazu Iwamura
Byung-Gyu Kim
author_sort Debashis Das Chakladar
title 3D Avatar Approach for Continuous Sign Movement Using Speech/Text
title_short 3D Avatar Approach for Continuous Sign Movement Using Speech/Text
title_full 3D Avatar Approach for Continuous Sign Movement Using Speech/Text
title_fullStr 3D Avatar Approach for Continuous Sign Movement Using Speech/Text
title_full_unstemmed 3D Avatar Approach for Continuous Sign Movement Using Speech/Text
title_sort 3d avatar approach for continuous sign movement using speech/text
publisher MDPI AG
series Applied Sciences
issn 2076-3417
publishDate 2021-04-01
description Sign language is a visual language for communication used by hearing-impaired people with the help of hand and finger movements. Indian Sign Language (ISL) is a well-developed and standard way of communication for hearing-impaired people living in India. However, other people who use spoken language always face difficulty while communicating with a hearing-impaired person due to lack of sign language knowledge. In this study, we have developed a 3D avatar-based sign language learning system that converts the input speech/text into corresponding sign movements for ISL. The system consists of three modules. Initially, the input speech is converted into an English sentence. Then, that English sentence is converted into the corresponding ISL sentence using the Natural Language Processing (NLP) technique. Finally, the motion of the 3D avatar is defined based on the ISL sentence. The translation module achieves a 10.50 SER (Sign Error Rate) score.
topic Indian Sign Language (ISL)
natural language processing
avatar
sign movement
context-free grammar
url https://www.mdpi.com/2076-3417/11/8/3439
work_keys_str_mv AT debashisdaschakladar 3davatarapproachforcontinuoussignmovementusingspeechtext
AT pradeepkumar 3davatarapproachforcontinuoussignmovementusingspeechtext
AT shubhammandal 3davatarapproachforcontinuoussignmovementusingspeechtext
AT parthapratimroy 3davatarapproachforcontinuoussignmovementusingspeechtext
AT masakazuiwamura 3davatarapproachforcontinuoussignmovementusingspeechtext
AT byunggyukim 3davatarapproachforcontinuoussignmovementusingspeechtext
_version_ 1721529533248569344