STOCHASTIC GRADIENT METHODS FOR UNCONSTRAINED OPTIMIZATION

• Main Authors: Nataša Krejić, Nataša Krklec Jerinkić
• Format: Article
• Language: English
• Published: Sociedade Brasileira de Pesquisa Operacional 2014-12-01
• Series: Pesquisa Operacional
• Subjects: unconstrained optimization; stochastic gradient; stochastic approximation; sample average approximation
• Online Access: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0101-74382014000300373&lng=en&tlng=en

This papers presents an overview of gradient based methods for minimization of noisy functions. It is assumed that the objective functions is either given with error terms of stochastic nature or given as the mathematical expectation. Such problems arise in the context of simulation based optimization. The focus of this presentation is on the gradient based Stochastic Approximation and Sample Average Approximation methods. The concept of stochastic gradient approximation of the true gradient can be successfully extended to deterministic problems. Methods of this kind are presented for the data fitting and machine learning problems.