ETD

• large-width limit
• neural network

The thesis studies the large-width limit of fully connected, feedforward and deep neural networks, whose weights are initialized either according to the usual Gaussian (or LeCun) initialization, or by assigning to each hidden neuron a nonnegative random variable that controls the variance of its outgoing weights. Under the “random variances” initialization, the weights are dependent and possibly heavy-tailed, which is often desirable. We study the large-width limit for both initializations, at the finite-dimensional and functional level. Under LeCun initialization the limits are Gaussian, whereas under the “random variances” initialization the limits are conditionally Gaussian. The proofs of the functional results represent the original part of the thesis and rely on a suitable variant of Kolmogorov's continuity theorem. Following the recent literature, under LeCun initialization, we provide an upper bound on the total variation distance between the finite-dimensional distributions of the output and the finite-dimensional distributions of the limiting Gaussian field. It is proved that, if the number of neurons in the hidden layers grows as n, then the total variation distance is O(1/n). We provide a similar behavior under the “random variances” initialization, for the shallow neural network. We illustrate the theoretical results with some simulations.