Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

A related viewpoint is that overparametrization is good because the model is stranded when the Hessian has all positive/zero eigenvalues. If we treat the probability that a particular Hessian eigenvalue turns positive as a Bernoulli process, the chance of all eigenvalues going positive/zero exponentially decreases as the parameter count increases [1] https://arxiv.org/abs/1406.2572

This paper makes the points that it's the saddles and not local minima that are the problem: https://arxiv.org/abs/1406.2572 It was the basis for adding 'momentum' to optimizers - so that you could skate across the saddles.

Maybe GP is talking about the hypothesis that in high-dimensional space you will probably almost always find /some/ direction which decreases the error function. IIRC this hypothesis is well supported, but not yet proven. So the real problem may not be local minima but saddle points that just /appear/ to be minima due to slow optimisation progress. For more details, check out https://arxiv.org/abs/1406.2572

See "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization" https://arxiv.org/abs/1406.2572 The 'bowl' you described are these saddle points, and this 2014 paper was pretty big for tackling it. But, as others pointed out Adam or such optimizers are used more often than steepest descent.

That's why I said high dimensional neural networks. There's been a lot of literature explaining why local minima aren't a problem in very high dimension loss surfaces. Check any of the literature on this subject: https://arxiv.org/abs/1611.06310v2 https://arxiv.org/abs/1406.2572 Local minima are something that people thought was gonna be a problem, especially back in the 2000s. They played around with small neural nets on toy examples such as yours, and thought it was intractable. It's the entire reason why neural nets fell out of the fashion in the early 2000s, and people moved towards techniques like SVM. These toy examples don't generalize to high dimensions, and if you take a look at the literature, you'll see that the consensus agrees with my statement.

Minima/maxima in high-dimensional spaces are not much of an issue; the bigger issue are saddle points. cf https://arxiv.org/pdf/1406.2572.pdf This is presuming that the problem has a continuous formulation anyways.

Your statement is true-ish, but not true enough. It is true there are more local minima, but they're generally pretty equivalent. In fact, in high dimensions, saddle points are a much bigger problem than local minima. [0] The fact that optimization works on these large, non-convex problems is kind of a marvel, and raises really interesting questions. (mostly "this shouldn't work this well unless there's some kind of regularity to these problems that makes them nicely behaved"). What this says about evolution I neither know nor care to speculate (above my pay grade). [0] http://arxiv.org/abs/1406.2572 Yes, the backpropaganda can get annoying.

Because our intuition about local minima is wrong in extremely high dimensional spaces. In two and three dimensions, local minima are common. In a million dimensions, local minima are rare. The intuitive explanation is that for a local minimum to exist, the function must be curving up (first derivative = 0, second derivative >= 0) simultaneously in every dimension. It makes sense that as you add more dimensions this becomes less and less likely, and for a million dimensions it's vanishingly unlikely. What you get instead of local minima are saddle points, where some dimensions are curving up and some are curving down. Saddle points can also be problematic for optimization but they can be dealt with using fancy optimization techniques. For a more rigorous explanation, see http://arxiv.org/abs/1406.2572