• stochastic
• gradient
• descent
• optimization
• algorithm
• adaptive
• asymptotic
• convergence
• learning rate
• batch
• momentum
• adagrad
• adadelta
• adam
• deep learning
• machine learning

Gradient-based optimization algorithms, in particular their stochastic counterparts, have become by far the most widely used algorithms in a lot of applied mathematics problems, first of all statistical modeling and machine learning. After a quick introduction to gradient descent and its stochastic counterpart, the focus is put on variance reduction techniques such as learning rate reduction, batch size increase and momentum. Results about optimal scheduling for these techniques are given in the case of convex loss functions. In the non-convex case, a description of adaptive descent algorithm is given, as well as some theoretical results about their convergence. The framework of stochastic modified equations, in which descent algorithms can be interpreted as discretizations of stochastic differential equations gets introduced and ultimately extended to include adaptive algorithms.