A Deep Learning Perspective on Beauty, Sentiment, and Remembrance of Art

• Main Authors: Eva Cetinic, Tomislav Lipic, Sonja Grgic
• Format: Article
• Language: English
• Published: IEEE 2019-01-01
• Series: IEEE Access
• Subjects: Convolutional neural networks; image aesthetics; image memorability; fine art; visual sentiment
• Online Access: https://ieeexplore.ieee.org/document/8731853/

With the emergence of large digitized fine art collections and the successful performance of deep learning techniques, new research prospects unfold in the intersection of artificial intelligence and art. In order to explore the applicability of deep learning techniques in understanding art images beyond object recognition and classification, we employ convolutional neural networks (CNN) to predict scores related to three subjective aspects of human perception: aesthetic evaluation of the image, sentiment evoked by the image, and memorability of the image. For each concept, we evaluate several different CNN models trained on various natural image datasets and select the best performing model based on the qualitative results and the comparison with existing subjective ratings of artworks. Furthermore, we employ different decision tree-based machine learning models to analyze the relative importance of various image features related to the content, composition, and color in determining image aesthetics, visual sentiment, and memorability scores. Our findings suggest that content and image lighting have significant influence on aesthetics, in which color vividness and harmony strongly influence sentiment prediction, while object emphasis has a high impact on memorability. In addition, we explore the predicted aesthetic, sentiment, and memorability scores in the context of art history by analyzing their distribution in regard to different artistic styles, genres, artists, and centuries. The presented approach enables new ways of exploring fine art collections based on highly subjective aspects of art, as well as represents one step forward toward bridging the gap between traditional formal analysis and the computational analysis of fine art.