Geometric Regularization of Local Activations for Knowledge Transfer in Convolutional Neural Networks

• Main Authors: Ilias Theodorakopoulos, Foteini Fotopoulou, George Economou
• Format: Article
• Language: English
• Published: MDPI AG 2021-08-01
• Series: Information
• Subjects: manifold regularization; knowledge transfer; knowledge distillation; deep learning with limited data
• Online Access: https://www.mdpi.com/2078-2489/12/8/333

In this work, we propose a mechanism for knowledge transfer between Convolutional Neural Networks via the geometric regularization of local features produced by the activations of convolutional layers. We formulate appropriate loss functions, driving a “student” model to adapt such that its local features exhibit similar geometrical characteristics to those of an “instructor” model, at corresponding layers. The investigated functions, inspired by manifold-to-manifold distance measures, are designed to compare the neighboring information inside the feature space of the involved activations without any restrictions in the features’ dimensionality, thus enabling knowledge transfer between different architectures. Experimental evidence demonstrates that the proposed technique is effective in different settings, including knowledge-transfer to smaller models, transfer between different deep architectures and harnessing knowledge from external data, producing models with increased accuracy compared to a typical training. Furthermore, results indicate that the presented method can work synergistically with methods such as knowledge distillation, further increasing the accuracy of the trained models. Finally, experiments on training with limited data show that a combined regularization scheme can achieve the same generalization as a non-regularized training with 50% of the data in the CIFAR-10 classification task.