The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Two thoughts:
> how important is computing the exact gradient using calculus
Normally the gradient is computed with a small "minibatch" of examples, meaning that on average over many steps the true gradient is followed, but each step individually never moves exacty along the true gradient. This noisy walk is actually quite beneficial for the final performance of the network https://arxiv.org/abs/2006.15081 , https://arxiv.org/abs/1609.04836 so much so that people started wondering what is the best way to "corrupt" this approximate gradient even more to improve performance https://arxiv.org/abs/2202.02831 (and many other works relating to SGD noise)
> vs just knowing the…
Read the full comment
general direction to step
I can't find relevant papers now, but I seem to recall that the Hessian eigenvalues of the loss function decay rather quickly, which means that taking a step in most directions will not change the loss very much. That is to say, you have to know which direction to go quite precisely for an SGD-like method to work. People have been trying to visualize the loss and trajectory taken during optimization https://arxiv.org/pdf/1712.09913 , https://losslandscape.com/
Sorry, my last sentence
> Otherwise, models can achieve very good generalization.
was probably unclear. When we talk about generalization, we typically have to assume a lack of distributional shift. Otherwise, if the "truly unseen data" really is drawn from a different distribution (say one where all cats are now called "dogs" and all dogs are now called "cats"), we can get arbitrarily poor performance.
As for the flat minima part, lemme pull up some papers from my notes.
https://arxiv.org/abs/1609.04836 showed larger batch sizes lead to sharper minima.
> ... and as is well known, sharp minima lead to poorer generalization.
https://arxiv.org/abs/1711.01530 provides a…
Read the full comment
parametrization-invariant notion of "flatness" that counters Dinh et al's point in https://arxiv.org/abs/1703.04933. I thought this paper looks pretty promising, although I haven't gone through the math.
>Although flatness of the loss function has been
linked to generalization [Hochreiter and Schmidhuber, 1997], existing definitions
of flatness are neither invariant to nodewise re-scalings of ReLU nets nor general
coordinate transformations [Dinh et al., 2017] of the parameter space, which calls
into question their utility for describing generalization
https://arxiv.org/abs/1703.11008 is also a really promising paper on generalization that basically shows another way to view flatness from a Bayesian perspective (they also show the most impressive generalization bound I've seen yet).
And of course, the classic Schmidhuber paper on Flat Minima: http://www.bioinf.jku.at/publications/older/3304.pdf
I think it's clear from these papers that the "flat minima generalize well" argument is a common avenue of research with respect to generalization. Many of the generalization papers I've read recently are mostly related to different ways of calculating flatness, and pretty much all of them make reference and link their work to the 1st and 4th paper I linked above.
> shouldn't a larger batch_size inherently provide more information about the optimal direction of the update?
Not necessarily, since a batch gradient output (as I understand it, and at least used to code it) all gets averaged together.
Consider standing in a valley with two equal hills either side of you. If you were to try one direction and see that climbing that way helps, you'd take a step that way. Then the next step would keep taking you up that hill.
Now, if you batched together two direction tests, what would happen? You'd average together your left and right and end up with moving nowhere. Having both at the same time doesn't give you better information about how…
Read the full comment
you move if you only see the result after averaging.
This interestingly maps to something we see in humans, though I'm struggling to find a decent paper on it (from the PRISM lab in Birmingham, UK if anyone else has any luck, think the person doing the research might have been called Chris). Simple adaptation tasks, in this case learning to control a joystick that has a clockwise/anticlockwise force applied to it, don't work well if you try and learn both one thing and the opposite straight away. However, sleeping in-between learning each left you able to do both well. Perhaps this was early results though.
Batch tradeoffs:
https://stats.stackexchange.com/questions/164876/tradeoff-ba...
https://arxiv.org/abs/1609.04836
My understanding from this article is that for a greater GPU ram size you will need to increase the minibatch size which has a negative impact on the models ability to generalize[0].
[0] https://arxiv.org/abs/1609.04836
Then the total batch size is growing with the number of GPUs and the convergence might be impacted both in terms of speed and solution quality (e.g. https://arxiv.org/abs/1609.04836 ).
I could believe you if tell you me that the validation loss and test accuracy of the large distributed model remains as good as the sequential, single GPU model after the same total number of epochs but this is not a given and if it's not the case I would find those benchmarks deceptive.
Really don't think that's the best paper to say "sheds quite a bit of light on this". That paper has been somewhat controversial since it came out.
I think https://arxiv.org/abs/1609.04836 is seminal in showing unsharp minima = generalization, the parent's paper is good for showing that gradient descent over non-convex surfaces works fine, https://arxiv.org/abs/1611.03530 is landmark for kicking off this whole generalization business (mainly shows that traditional models of generalization, namely VC dimension and ideas of "capacity" don't make sense for neural nets).
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.