ETD

Archivio digitale delle tesi discusse presso l'Università di Pisa

Tesi etd-08262022-123703


Tipo di tesi
Tesi di laurea magistrale
Autore
ARNABOLDI, LUCA
URN
etd-08262022-123703
Titolo
Learning transitions for one-pass stochastic gradient descent on shallow neural networks
Dipartimento
FISICA
Corso di studi
FISICA
Relatori
relatore Prof. Krzakala, Florent
relatore Dott. Loureiro, Bruno
Parole chiave
  • machine learning
  • neural networks
  • statistical physics
  • phase retrieval
Data inizio appello
14/09/2022
Consultabilità
Tesi non consultabile
Riassunto
In recent years, neural networks have made possible great progress in several fields of artificial intelligence, but their theoretical understanding is still lacking. In this thesis, we study the high-dimensional input limit of a two-layer neural network, through statistical physic tools. Using the squared activation function we are able to derive some ODEs for the dynamics of sufficient statistics, that can then be used for estimating time of transition between learning phases. We apply this analysis to the simplest case known as phase retrieval, exploring different kinds of initial conditions. We then study the dynamics with the weights constrained on a hypersphere; we estimate the exit time from the first phase of learning, therefrom we derive an estimate of the gain that occurs by overparameterizing the network. We conclude by adding a stochastic corrective term to the equations, showing that this leads to a better estimation of the exit times.
File