A Multimodal System Towards Intention Anticipation and Missing Actions Reminder via Continuous Self-Learning

• Main Authors: Chien, Ting-An, 簡廷安
• Other Authors: Sun, Min
• Format: Others
• Language: en_US
• Published: 2018
• Online Access: http://ndltd.ncl.edu.tw/handle/67apg2

碩士 === 國立清華大學 === 電機工程學系所 === 106 === We imagine a future smart home full of Internet-of-Thing Devices (IoT-D) with the capability to log and trigger events (e.g., brewing coffee and turn-on TV). A novel multimodal system is proposed to anticipate user intention (IoT-D ON states) and remind missing actions (IoT-D OFF states) via continuous learning such that IoT-D state changes can be automated in future smart home (e.g., turn on TV automatically). Our system fuses synchronized IoT-D states (i.e., TV ON or OFF), wearable sensors (wrist-mounted camera and accelerometer), and 360$^{\circ}$ overhead motion vectors to incorporate environment condition, human affordance (activity and interaction with objects), and user position, respectively. Our core contribution is an end-to-end trainable hierarchical Recurrent Neural Network (RNN) with late fusion to (1) classify intention into none, long-term, mid-term, short-term, immediate intentions, and (2) classify none or missing actions. It can automatically (without human annotation) adapt to user's behavior by continuous learning using synchronized IoT-D states and sensors data. Moreover, we introduce a countdown loss for training anticipation and further improve our model performance. We also learn to remind given ZERO missing actions in training using our data augmentation procedure. Our system is evaluated on an in-house smart home environment and achieves not only reasonably good accuracy but also shown its ability to adapt user behavior variation and generalization across users.