Reconciling modern machine learning practice and the bias-variance trade-off

And yet that is not what is observed in practice. See figures 1 and 2 of that paper [1]. What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply). [1] https://arxiv.org/pdf/1812.11118.pdf

Large model capacity enough to perfectly memorize/interpolate data may not be a bad thing. A phenomenon known as "deep double descent" says that increasing modeling capacity relative to the dataset size can reduce generalization error, even after the model achieves perfect training performance (see work by Mikhail Belkin [0] and empirical demonstrations on large deep learning tasks by researchers from Harvard/OpenAI [1]). Other work argues that memorization is critical to good performance on real-world tasks where the data distribution is often long-tailed [2]: to perform well at tasks where the training data set only has 1 or 2 examples, it's best to memorize those labels, rather than extrapolate from other data (which a lower capacity model may prefer to do). [0]: https://arxiv.org/abs/1812.11118 [1]: https://openai.com/blog/deep-double-descent/ [2]: https://arxiv.org/abs/2008.03703

TL;DR: For high-dimensional models (say, with millions to billions of parameters), there's always a good set parameters nearby, and when we start descending towards it, we are highly unlikely to get stuck, because almost always there's at least one path down along at least one among of all those dimensions -- i.e., there are no local optima. Once we've stumbled upon a good set of parameters, as measured by validation, we can stop. These intuitions are consistent with my experience... but I think there's more to deep learning. For instance, these intuitions fail to explain "weird" phenomena, such as "double descent" and "interpolation thresholds": * https://openai.com/blog/deep-double-descent/ * https://arxiv.org/abs/1809.09349 * https://arxiv.org/abs/1812.11118 * See also: http://www.stat.cmu.edu/~ryantibs/papers/lsinter.pdf We still don't fully understand why stochastic gradient descent works so well in so many domains.