Multimodal Emotion Recognition from Art Using Sequential Co-Attention

Multimodal Emotion Recognition from Art Using Sequential Co-Attention

In this study, we present a multimodal emotion recognition architecture that uses both feature-level attention (sequential co-attention) and modality attention (weighted modality fusion) to classify emotion in art. The proposed architecture helps the model to focus on learning informative and refine...

Full description

Bibliographic Details
Main Authors: Tsegaye Misikir Tashu, Sakina Hajiyeva, Tomas Horvath
Format: Article
Language:English
Published: MDPI AG 2021-08-01
Series:Journal of Imaging
Subjects:
multimodal
emotions
attention
art
modality fusion
emotion analysis
Online Access:https://www.mdpi.com/2313-433X/7/8/157
id doaj-6a463e1b0b374401b00de8db4c593bbd
record_format Article
spelling doaj-6a463e1b0b374401b00de8db4c593bbd2021-08-26T13:56:37ZengMDPI AGJournal of Imaging2313-433X2021-08-01715715710.3390/jimaging7080157Multimodal Emotion Recognition from Art Using Sequential Co-AttentionTsegaye Misikir Tashu0Sakina Hajiyeva1Tomas Horvath2Department of Data Science and Engineering (T-Labs), Faculty of Informatics, Eötvös Loránd University, Pázmány Péter Sétány 1/C, 1117 Budapest, HungaryDepartment of Data Science and Engineering (T-Labs), Faculty of Informatics, Eötvös Loránd University, Pázmány Péter Sétány 1/C, 1117 Budapest, HungaryDepartment of Data Science and Engineering (T-Labs), Faculty of Informatics, Eötvös Loránd University, Pázmány Péter Sétány 1/C, 1117 Budapest, HungaryIn this study, we present a multimodal emotion recognition architecture that uses both feature-level attention (sequential co-attention) and modality attention (weighted modality fusion) to classify emotion in art. The proposed architecture helps the model to focus on learning informative and refined representations for both feature extraction and modality fusion. The resulting system can be used to categorize artworks according to the emotions they evoke; recommend paintings that accentuate or balance a particular mood; search for paintings of a particular style or genre that represents custom content in a custom state of impact. Experimental results on the WikiArt emotion dataset showed the efficiency of the approach proposed and the usefulness of three modalities in emotion recognition.https://www.mdpi.com/2313-433X/7/8/157multimodalemotionsattentionartmodality fusionemotion analysis
collection DOAJ
language English
format Article
sources DOAJ
author Tsegaye Misikir Tashu
Sakina Hajiyeva
Tomas Horvath
spellingShingle Tsegaye Misikir Tashu
Sakina Hajiyeva
Tomas Horvath
Multimodal Emotion Recognition from Art Using Sequential Co-Attention
Journal of Imaging
multimodal
emotions
attention
art
modality fusion
emotion analysis
author_facet Tsegaye Misikir Tashu
Sakina Hajiyeva
Tomas Horvath
author_sort Tsegaye Misikir Tashu
title Multimodal Emotion Recognition from Art Using Sequential Co-Attention
title_short Multimodal Emotion Recognition from Art Using Sequential Co-Attention
title_full Multimodal Emotion Recognition from Art Using Sequential Co-Attention
title_fullStr Multimodal Emotion Recognition from Art Using Sequential Co-Attention
title_full_unstemmed Multimodal Emotion Recognition from Art Using Sequential Co-Attention
title_sort multimodal emotion recognition from art using sequential co-attention
publisher MDPI AG
series Journal of Imaging
issn 2313-433X
publishDate 2021-08-01
description In this study, we present a multimodal emotion recognition architecture that uses both feature-level attention (sequential co-attention) and modality attention (weighted modality fusion) to classify emotion in art. The proposed architecture helps the model to focus on learning informative and refined representations for both feature extraction and modality fusion. The resulting system can be used to categorize artworks according to the emotions they evoke; recommend paintings that accentuate or balance a particular mood; search for paintings of a particular style or genre that represents custom content in a custom state of impact. Experimental results on the WikiArt emotion dataset showed the efficiency of the approach proposed and the usefulness of three modalities in emotion recognition.
topic multimodal
emotions
attention
art
modality fusion
emotion analysis
url https://www.mdpi.com/2313-433X/7/8/157
work_keys_str_mv AT tsegayemisikirtashu multimodalemotionrecognitionfromartusingsequentialcoattention
AT sakinahajiyeva multimodalemotionrecognitionfromartusingsequentialcoattention
AT tomashorvath multimodalemotionrecognitionfromartusingsequentialcoattention
_version_ 1721192256843546624

Similar Items