Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition
Spontaneous speech emotion recognition is a new and challenging research topic. In this paper, we propose a new method of spontaneous speech emotion recognition on the basis of binaural representations and deep convolutional neural networks (CNNs). The proposed method initially employs multiple CNNs...
Main Authors: | , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2020-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/8967041/ |
id |
doaj-1a822beaa54c48608406e752862bdd2d |
---|---|
record_format |
Article |
spelling |
doaj-1a822beaa54c48608406e752862bdd2d2021-03-30T01:14:46ZengIEEEIEEE Access2169-35362020-01-018234962350510.1109/ACCESS.2020.29690328967041Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion RecognitionShiqing Zhang0https://orcid.org/0000-0001-8184-5088Aihua Chen1Wenping Guo2https://orcid.org/0000-0002-0405-1775Yueli Cui3https://orcid.org/0000-0002-9837-6705Xiaoming Zhao4https://orcid.org/0000-0002-4708-4171Limei Liu5https://orcid.org/0000-0003-0105-074XInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Intelligent Information Processing, Taizhou University, Taizhou, ChinaInstitute of Big Data and Internet Innovation, Hunan University of Commerce, Changsha, ChinaSpontaneous speech emotion recognition is a new and challenging research topic. In this paper, we propose a new method of spontaneous speech emotion recognition on the basis of binaural representations and deep convolutional neural networks (CNNs). The proposed method initially employs multiple CNNs to learn deep segment-level binaural representations such as Left-Right and Mid-Side pairs from the extracted image-like Mel-spectrograms. These CNNs are fine-tuned on target emotional speech datasets from a pre-trained image CNN model. Then, a new feature pooling strategy, called block-based temporal feature pooling, is proposed to aggregate the learned segment-level features for producing fixed-length utterance-level features. Based on the utterance-level features, linear support vector machines (SVM) is adopted for emotion classification. Finally, a two-stage score-level fusion strategy is used to integrate the obtained results from Left-Right and Mid-Side pairs. Extensive experiments on two challenging spontaneous emotional speech datasets, including the AFEW5.0 and BAUM-1s databases, demonstrate the effectiveness of our proposed method.https://ieeexplore.ieee.org/document/8967041/Spontaneous speech emotion recognitionbinaural representationsdeep convolutional neural networkstemporal feature pooling |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Shiqing Zhang Aihua Chen Wenping Guo Yueli Cui Xiaoming Zhao Limei Liu |
spellingShingle |
Shiqing Zhang Aihua Chen Wenping Guo Yueli Cui Xiaoming Zhao Limei Liu Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition IEEE Access Spontaneous speech emotion recognition binaural representations deep convolutional neural networks temporal feature pooling |
author_facet |
Shiqing Zhang Aihua Chen Wenping Guo Yueli Cui Xiaoming Zhao Limei Liu |
author_sort |
Shiqing Zhang |
title |
Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition |
title_short |
Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition |
title_full |
Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition |
title_fullStr |
Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition |
title_full_unstemmed |
Learning Deep Binaural Representations With Deep Convolutional Neural Networks for Spontaneous Speech Emotion Recognition |
title_sort |
learning deep binaural representations with deep convolutional neural networks for spontaneous speech emotion recognition |
publisher |
IEEE |
series |
IEEE Access |
issn |
2169-3536 |
publishDate |
2020-01-01 |
description |
Spontaneous speech emotion recognition is a new and challenging research topic. In this paper, we propose a new method of spontaneous speech emotion recognition on the basis of binaural representations and deep convolutional neural networks (CNNs). The proposed method initially employs multiple CNNs to learn deep segment-level binaural representations such as Left-Right and Mid-Side pairs from the extracted image-like Mel-spectrograms. These CNNs are fine-tuned on target emotional speech datasets from a pre-trained image CNN model. Then, a new feature pooling strategy, called block-based temporal feature pooling, is proposed to aggregate the learned segment-level features for producing fixed-length utterance-level features. Based on the utterance-level features, linear support vector machines (SVM) is adopted for emotion classification. Finally, a two-stage score-level fusion strategy is used to integrate the obtained results from Left-Right and Mid-Side pairs. Extensive experiments on two challenging spontaneous emotional speech datasets, including the AFEW5.0 and BAUM-1s databases, demonstrate the effectiveness of our proposed method. |
topic |
Spontaneous speech emotion recognition binaural representations deep convolutional neural networks temporal feature pooling |
url |
https://ieeexplore.ieee.org/document/8967041/ |
work_keys_str_mv |
AT shiqingzhang learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition AT aihuachen learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition AT wenpingguo learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition AT yuelicui learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition AT xiaomingzhao learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition AT limeiliu learningdeepbinauralrepresentationswithdeepconvolutionalneuralnetworksforspontaneousspeechemotionrecognition |
_version_ |
1724187334370394112 |