We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model under the assumptions of: i) variable independence, ii) redundancy in network parametrization, and iii) uniformity. These assumptions enable us to explain the complexity of the fully decoupled neural network through the prism of the results from random matrix theory. We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum. The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor quality local minima have non-zero probability of being recovered. Finally, we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
There is a saying in mathematics that the fastest way to a solution is through the complex plane. This was discovered because a lot of proofs are nicer by doing analytical continuation and analyzing the properties of the continued function. Complex-step differentiation is another example of this.
In some sense, something similar applies to neural networks in this context. Have you done a lot of fitting of classical basis methods inside of differential equations? They are very prone to local minima, so direct training of polynomials inside of a differential equation is rather hard. But through neural network magic, somehow related to [1], which essentially state that local…
Read the full comment
minima are the global minima on large enough neural networks. So this lets you get pretty lazy and just do local optimization to find missing functions, and then sparsify to polynomials later, in a way where the optimization is better behaved than going directly to polynomials. The DiffEqFlux library has both approaches available, so you can try both side by side and see the difference. From years of experience doing the former, the latter is quite a breath of fresh air.
[1] https://arxiv.org/abs/1412.0233
i suspect that what he's referring to is that he's heuristically minimizing a somewhat arbitrary (loss) function in a million-ish dimensions using the simple variants of gradient descent that work under these conditions. it sounds far too WIBNI to produce good results reliably (in practice, let alone in theory). the landscape has so many stationary points at which to get stuck; why would you ever get good results?
there's a small cottage industry of papers (like [0]) that try to explain this.
[0] https://arxiv.org/pdf/1412.0233.pdf
Analogously Neural Network learning, i.e. learning modeled as a phase transition between disorder and order:
“... studies the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model”
from "The Loss Surfaces of Multilayer Networks"
Choromanska, Henaff, Mathieu, Arous, & LeCun [1]
[1] https://arxiv.org/abs/1412.0233
Also the physics of collapsing sand piles.
If you mean the lack of theoretical guarantees due to a highly non-convex optimization, you could argue the same thing about 30 year old 2-layer networks. And we actually do have a decent idea why deep learning works well:
http://arxiv.org/pdf/1412.6544v4.pdf
http://arxiv.org/pdf/1412.0233.pdf
I believe that has to do with the vanishing gradient issue which has been studied.
As for why nets converge at all, there's this paper whichI believe tries to establish some theory why bigger nets don't have such a big problem with local optima: http://arxiv.org/abs/1412.0233
This is as good a post as any about the theory of why deep learning works so well - esp. if you know a little about spin models from statistical mechanics
https://charlesmartin14.wordpress.com/2015/02/26/why-does-de...
additional flavor: http://arxiv.org/pdf/1412.0233.pdf
and slides 16-22 of https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsd...
>an infinitely deep net (whatever that means) would not generalize on little data, and would get even harder to train the deeper it gets.
Not with proper priors/regularization.
>You've argued some ideas from intuition. There is little theoretical rigor around this, however.
There's this paper which goes more into theoretical depth on the idea: http://arxiv.org/abs/1412.0233
http://arxiv.org/abs/1412.0233
>The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor quality local minima have non-zero probability of being recovered.
As a counterpoint, the Blessing of Dimensionality is that local minima are mostly as good as the global minima.
In David Silver's RLDM 2015 talk @20 mins in
http://videolectures.net/rldm2015_silver_reinforcement_learn...
he recounts this as his experience.
the paper The Loss Surfaces of Multilayer Networks goes in depth
http://arxiv.org/abs/1412.0233
[edit] and the paper
Qualitative Characterisation of Neural Net Optimisation
http://arxiv.org/abs/1412.6544v4
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.