Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition

Spontaneous speech emotion recognition is a new and challenging research topic. In this paper, we propose a new method of spontaneous speech emotion recognition on the basis of binaural representations and deep convolutional neural networks (CNNs). The proposed method initially employs multiple CNNs...

Full description

Bibliographic Details
Main Authors: Shiqing Zhang, Aihua Chen, Wenping Guo, Yueli Cui, Xiaoming Zhao, Limei Liu
Format: Article
Language:English
Published: IEEE 2020-01-01
Series:IEEE Access
Subjects:
Online Access:https://ieeexplore.ieee.org/document/8967041/
id doaj-1a822beaa54c48608406e752862bdd2d
record_format Article
spelling doaj-1a822beaa54c48608406e752862bdd2d2021-03-30T01:14:46ZengIEEEIEEE Access2169-35362020-01-018234962350510.1109/ACCESS.2020.29690328967041Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion RecognitionShiqing Zhang0https://orcid.org/0000-0001-8184-5088Aihua Chen1Wenping Guo2https://orcid.org/0000-0002-0405-1775Yueli Cui3https://orcid.org/0000-0002-9837-6705Xiaoming Zhao4https://orcid.org/0000-0002-4708-4171Limei Liu5https://orcid.org/0000-0003-0105-074XInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Big Data and Internet Innovation, Hunan University of Commerce, Changsha, ChinaSpontaneous speech emotion recognition is a new and challenging research topic. In this paper, we propose a new method of spontaneous speech emotion recognition on the basis of binaural representations and deep convolutional neural networks (CNNs). The proposed method initially employs multiple CNNs to learn deep segment-level binaural representations such as Left-Right and Mid-Side pairs from the extracted image-like Mel-spectrograms. These CNNs are fine-tuned on target emotional speech datasets from a pre-trained image CNN model. Then, a new feature pooling strategy, called block-based temporal feature pooling, is proposed to aggregate the learned segment-level features for producing fixed-length utterance-level features. Based on the utterance-level features, linear support vector machines (SVM) is adopted for emotion classification. Finally, a two-stage score-level fusion strategy is used to integrate the obtained results from Left-Right and Mid-Side pairs. Extensive experiments on two challenging spontaneous emotional speech datasets, including the AFEW5.0 and BAUM-1s databases, demonstrate the effectiveness of our proposed method.https://ieeexplore.ieee.org/document/8967041/Spontaneous speech emotion recognitionbinaural representationsdeep convolutional neural networkstemporal feature pooling
collection DOAJ
language English
format Article
sources DOAJ
author Shiqing Zhang
Aihua Chen
Wenping Guo
Yueli Cui
Xiaoming Zhao
Limei Liu
spellingShingle Shiqing Zhang
Aihua Chen
Wenping Guo
Yueli Cui
Xiaoming Zhao
Limei Liu
Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition
IEEE Access
Spontaneous speech emotion recognition
binaural representations
deep convolutional neural networks
temporal feature pooling
author_facet Shiqing Zhang
Aihua Chen
Wenping Guo
Yueli Cui
Xiaoming Zhao
Limei Liu
author_sort Shiqing Zhang
title Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition
title_short Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition
title_full Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition
title_fullStr Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition
title_full_unstemmed Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition
title_sort learning deep binaural representations with deep convolutional neural networks for spontaneous speech emotion recognition
publisher IEEE
series IEEE Access
issn 2169-3536
publishDate 2020-01-01
description Spontaneous speech emotion recognition is a new and challenging research topic. In this paper, we propose a new method of spontaneous speech emotion recognition on the basis of binaural representations and deep convolutional neural networks (CNNs). The proposed method initially employs multiple CNNs to learn deep segment-level binaural representations such as Left-Right and Mid-Side pairs from the extracted image-like Mel-spectrograms. These CNNs are fine-tuned on target emotional speech datasets from a pre-trained image CNN model. Then, a new feature pooling strategy, called block-based temporal feature pooling, is proposed to aggregate the learned segment-level features for producing fixed-length utterance-level features. Based on the utterance-level features, linear support vector machines (SVM) is adopted for emotion classification. Finally, a two-stage score-level fusion strategy is used to integrate the obtained results from Left-Right and Mid-Side pairs. Extensive experiments on two challenging spontaneous emotional speech datasets, including the AFEW5.0 and BAUM-1s databases, demonstrate the effectiveness of our proposed method.
topic Spontaneous speech emotion recognition
binaural representations
deep convolutional neural networks
temporal feature pooling
url https://ieeexplore.ieee.org/document/8967041/
work_keys_str_mv AT shiqingzhang learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition
AT aihuachen learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition
AT wenpingguo learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition
AT yuelicui learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition
AT xiaomingzhao learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition
AT limeiliu learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition
_version_ 1724187334370394112