Automatic Evaluation of Recommendation Models

• Main Authors: Olga Alieva, Elena Gangan, Eugene Ilyushin, Alexey Kachalin
• Format: Article
• Language: Russian
• Published: The Fund for Promotion of Internet media, IT education, human development «League Internet Media» 2020-09-01
• Series: Современные информационные технологии и IT-образование
• Subjects: automatization; collaborative filtering; recommender systems; recommender tool
• Online Access: http://sitito.cs.msu.ru/index.php/SITITO/article/view/656

The paper presents an overview of state-of-the-art algorithms used in recommender systems. We discuss the goal of collaborative filtering (CF) as well as different approaches to the method. Specifically, we talk about Singular Value Decomposition (including optimizations, bias, time sensitive Singular Value Decomposition (SVD) and enhanced SVD methods as SVD++), clustering approaches (using K means clustering). We also discuss deep learning methods applied to recommender systems, such as Autoencoders and Restricted Boltzmann Machines. We also go through qualitative evaluation metrics of the algorithms, with a special emphasis on the classification quality metrics, as recommender systems are usually expected to have an order in which the recommendations are delivered. At the same time, we propose a tool that automates the processes of CF algorithms launch and evaluation, that contains data pre-processing, metrics selection, training launch, quality indicators checks and analyses of the resulted data. Our tool demonstrates the impact that parameter selection has on the quality of the algorithm execution. We observed that classical matrix factorization algorithms can compete with new deep learning methods, giving the correct tuning. Also, we demonstrate a significant gain in time between the manual (involving a person that launches all the algorithms individually) and the automatic (when the tool launches all the algorithms) algorithm launch.