Combining heterogeneous inputs for the development of adaptive and multimodal interaction systems

• Main Authors: David GRIOL, Jesús GARCÍA-HERRERO, José Manuel MOLINA
• Format: Article
• Language: English
• Published: Ediciones Universidad de Salamanca 2013-11-01
• Series: Advances in Distributed Computing and Artificial Intelligence Journal
• Subjects: software agents; multimodalfusion; visualsensornetworks; surveillance applications; spoken interaction; conversational agents; user modeling; dialog management
• Online Access: https://revistas.usal.es/index.php/2255-2863/article/view/11279

<p><em>In this paper we present a novel framework for the integration of visual sensor networks and speech-based interfaces. Our proposal follows the standard reference architecture in fusion systems (JDL), and combines different techniques related to Artificial Intelligence, Natural Language Processing and User Modeling to provide an enhanced interaction with their users. Firstly, the framework integrates a Cooperative Surveillance Multi-Agent System (CS-MAS), which includes several types of autonomous agents working in a coalition to track and make inferences on the positions of the targets. Secondly, enhanced conversational agents facilitate human-computer interaction by means of speech interaction. Thirdly, a statistical methodology allows modeling the user conversational behavior, which is learned from an initial corpus and improved with the knowledge acquired from the successive interactions. A technique is proposed to facilitate the multimodal fusion of these information sources and consider the result for the decision of the next system action.</em></p>