Deep learning of mean field annealing and gradient descent methods

• Main Authors: JIA-WEI JIEN, 簡嘉緯
• Other Authors: Jiann-Ming Wu
• Format: Others
• Published: 2017
• Online Access: http://ndltd.ncl.edu.tw/handle/85527454990153589583

碩士 === 國立東華大學 === 應用數學系 === 105 === Deep neural networks are widely used in machine learning. The deep neural network contains many hidden layers, with the powerful ability of corresponding input to output, through the adjustable neural interconnections. Deep neural network in different artificial intelligence framework, plays a different role, including feature extraction, dimensionality reduction, function approximation, and so on. The existing method of deep learning have been combined with a variety of methods, such as gradient descent, mini-batch, dropout, momentum, etc. Hinton’s two-stage learning method has been successfully applied in image recognition and speech recognition. The first stage is a restricted Boltzmann machine, RBMs[1], and the second stage is backpropagation learning[2] . The first stage of RBM learning gives a better initial parameter for the deep neural network. The second stage of learning is based on the initial parameters obtained in the first stage to optimize the built-in parameters of the neural network. This paper mainly proposes a new method of deep learning. The purpose is to avoid using the restricted Boltzmann method, rewrite Hinton’s two-stage method in one-stage method, and effectively achieve the goal of reducing mean square error and training error. The new method is based on the hybrid mean field annealing and gradient descent learning. In the S-type activation function, a parameter β represents the reciprocal of the annealing temperature. When the degree of entropy is greater, β value is smaller. When the degree of entropy is smaller, β value is greater. Let β grow from small to large gradually. The process is known as the annealing process. The optimal parameters in the process of annealing are recorded. The new method can be used to strength the learning ability of deep neural network, but also can combine the existing method, such as mini-batch, dropout, momentum to improve efficiency of learning. In this paper, handwriting recognition as an example is resolved to illustrate the effectiveness of new method.