Dialogue Quality and Nugget Detection for Short Text Conversation based on Hierarchical Multi-Stack Model with Memory Enhance Structure

• Main Authors: Hsiang-En Cherng, 程祥恩
• Other Authors: 張嘉惠
• Format: Others
• Language: en_US
• Published: 2019
• Online Access: http://ndltd.ncl.edu.tw/handle/53e785

碩士 === 國立中央大學 === 資訊工程學系 === 107 === With the development of Natural Language Processing (NLP) Automatic question-answering system such as Waston, Siri, Alexa, has become one of the most important NLP applications. Nowadays, enterprises try to build automatic custom service chatbots to save human resources and provide a 24-hour customer service. However, evaluation of chatbots currently relied greatly on human annotation which cost a plenty of time. Thus, Short Text Conversation 3 (STC-3) in NTCIR-14 has initiated a new subtask called Dialogue Quality (DQ) and Nugget Detection (ND) which aim to automatically evaluate dialogues generated by chatbots. In this paper, we consider the DQ and ND subtasks for STC-3 using deep learning method. The DQ subtask aims to judge the quality of the whole dialogue using three measures: Task Accomplishment (A-score), Dialogue Effectiveness (E-score) and Customer Satisfaction of the dialogue (S-score). The ND subtask, on the other hand, is to classify if an utterance in a dialogue contains a nugget, which is similar to dialogue act (DA) labeling problem. We applied a general model with utterance layer, context layer and memory layer to learn dialogue representation for both DQ and ND subtasks and use gating and attention mechanism at multiple layers including: utterance layer and context layer. We also tried BERT and multi-stack CNN as sentence representation. The result shows that BERT produced a better utterance representation than multi-stack CNN for both DQ and ND subtasks and outperform other participants’ model and the baseline models proposed by the organizer on Ubuntu customer helpdesk dialogues corpus.