Qualitatively characterizing neural network optimization problems

In very high dimensional spaces (like trying to optimize a neural network with billions of parameters), to be "in a valley", you must be in a valley with respect to every one of the billions of dimensions. It turns out that in many practical settings, loss landscapes are pretty well-behaved in this regard, and you can almost always find some direction to continue going downward in that lets you go around the next hill rather than over it. This 2015 paper has some good examples (although it does sort of sweep some issues under the rug): https://arxiv.org/pdf/1412.6544

If you mean the lack of theoretical guarantees due to a highly non-convex optimization, you could argue the same thing about 30 year old 2-layer networks. And we actually do have a decent idea why deep learning works well: http://arxiv.org/pdf/1412.6544v4.pdf http://arxiv.org/pdf/1412.0233.pdf

And yet Goodfellow et al's paper only uses 2 hidden layers and 1 softmax output layer (on MNIST) which is not so very deep but the effect is still present. (Qualitative Characterisation of Neural Net Optimisation ICML 2015) http://arxiv.org/abs/1412.6544v4 On p.6 Goodfellow et al. compare the neural nets to linear solutions by removing the non-linear layers, so that a single linear function is learned and find for Deep Linear Networks that : " All minima are global minima, and are linked to each other in a continuous manifold. " Which is pretty exciting because the received wisdom was local-minima are a really big problem.

As a counterpoint, the Blessing of Dimensionality is that local minima are mostly as good as the global minima. In David Silver's RLDM 2015 talk @20 mins in http://videolectures.net/rldm2015_silver_reinforcement_learn... he recounts this as his experience. the paper The Loss Surfaces of Multilayer Networks goes in depth http://arxiv.org/abs/1412.0233 [edit] and the paper Qualitative Characterisation of Neural Net Optimisation http://arxiv.org/abs/1412.6544v4