On attaining user-friendly hand gesture interfaces to control existing GUIs

• Main Authors: Egemen Ertugrul, Ping Li, Bin Sheng
• Format: Article
• Language: English
• Published: KeAi Communications Co., Ltd. 2020-04-01
• Series: Virtual Reality & Intelligent Hardware
• Subjects: Human-computer interaction; Gesture recognition; Computer vision applications
• Online Access: http://www.sciencedirect.com/science/article/pii/S2096579620300176

Background: Hand gesture interfaces are dedicated programs that principally perform hand tracking and hand gesture prediction to provide alternative controls and interaction methods. They take advantage of one of the most natural ways of interaction and communication, proposing novel input and showing great potential in the field of the human-computer interaction. Developing a flexible and rich hand gesture interface is known to be a time-consuming and arduous task. Previously published studies have demonstrated the significance of the finite-state-machine (FSM) approach when mapping detected gestures to GUI actions. Methods: In our hand gesture interface, we broadened the FSM approach by utilizing gesture-specific attributes, such as distance between hands, distance from the camera, and time of occurrences, to enable users to perform unique GUI actions. These attributes are obtained from hand gestures detected by the RealSense SDK employed in our hand gesture interface. By means of these gesture-specific attributes, users can activate static gestures and perform them as dynamic gestures. We also provided supplementary features to enhance the efficiency, convenience, and user-friendliness of our hand gesture interface. Moreover, we developed a complementary application for recording hand gestures by capturing hand keypoints in depth and color images to facilitate the generation of hand gesture datasets. Results: We conducted a small-scale user survey with fifteen subjects to test and evaluate our hand gesture interface. Anonymous feedback obtained from the users indicates that our hand gesture interface is adequately facile and self-explanatory to use. In addition, we received constructive feedback about minor flaws regarding the responsiveness of the interface. Conclusions: We proposed a hand gesture interface along with key concepts to attain user-friendliness and effectiveness in the control of existing GUIs.