Mapping an Auditory Scene Using Eye Tracking Glasses

• Main Authors: Fredriksson, Alfred, Wallin, Joakim
• Format: Others
• Language: English
• Published: Linköpings universitet, Reglerteknik 2020
• Subjects: Sensor Fusion; Engineering and Technology; Teknik och teknologier
• Online Access: http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-170849

The cocktail party problem introduced in 1953 describes the ability to focus auditory attention in a noisy environment epitomised by a cocktail party. An individual with normal hearing uses several cues to unmask talkers of interest, such cues often lacks for people with hearing loss. This thesis explores the possibility to use a pair of glasses equipped with an inertial measurement unit (IMU), monocular camera and eye tacker to estimate an auditory scene and estimate the attention of the person wearing the glasses. Three main areas of interest have been investigated: estimating head orientation of the user; track faces in the scene and determine talker of interest using gaze. Implemented on a hearing aid, this solution could be used to artificially unmask talkers in a noisy environment. The head orientation of the user has been estimated with an extended Kalman filter (\EKF) algorithm, with a constant velocity model and different sets of measurements: accelerometer; gyrosope; monocular visual odometry (MVO); gaze estimated bias (GEB). An intrinsic property of IMU sensors is a drift in yaw. A method using eye data and gyroscope measurements to estimate gyroscope bias has been investigated and is called GEB. The MVO methods investigated use either optical flow to track features in succeeding frames or a key frame approach to match features over multiple frames.Using estimated head orientation and face detection software, faces have been tracked since they can be assumed as regions of interest in a cocktail party environment. A constant position EKF with a nearest neighbour approach has been used for tracking. Further, eye data retrieved from the glasses has been analyzed to investigate the relation between gaze direction and current talker during conversations.