Skip to content
The methodOpen data
Back to the collection

Visualizing the Loss Landscape of Neural Nets

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein

2017 · arXiv

Machine learningHuman–computer interaction
Read the paper
HN score
0
Mentions
7
Readers
7
Months
7
What’s in a score?

From the abstract

Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

blackbear_
Neutral mention · rule
Two thoughts: > how important is computing the exact gradient using calculus Normally the gradient is computed with a small "minibatch" of examples, meaning that on average over many steps the true gradient is followed, but each step individually never moves exacty along the true gradient. This noisy walk is actually quite beneficial for the final performance of the network https://arxiv.org/abs/2006.15081 , https://arxiv.org/abs/1609.04836 so much so that people started wondering what is the best way to "corrupt" this approximate gradient even more to improve performance https://arxiv.org/abs/2202.02831 (and many other works relating to SGD noise) > vs just knowing the…
Read the full comment
general direction to step I can't find relevant papers now, but I seem to recall that the Hessian eigenvalues of the loss function decay rather quickly, which means that taking a step in most directions will not change the loss very much. That is to say, you have to know which direction to go quite precisely for an SGD-like method to work. People have been trying to visualize the loss and trajectory taken during optimization https://arxiv.org/pdf/1712.09913 , https://losslandscape.com/
Original comment Backpropagation is a leaky abstraction (2016)
dontwearitout
Neutral mention · rule
I haven't heard the term "shallow basin hypothesis" but I know what it refers to, these two papers spring to mind for me: 1) Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs https://arxiv.org/abs/1802.10026 2) Visualizing the Loss Landscape of Neural Nets https://arxiv.org/abs/1712.09913 There's also a very interesting body of work on merging trained models, such as by interpolating between points in weight space, which relates to the concept of "basins" of similar solutions. Skim the intro of this if you're interested in learning more: https://arxiv.org/abs/2211.08403
Original comment Grokfast: Accelerated Grokking by Amplifying Slow Gradients
Imnimo
Neutral mention · rule
Yes. To be more specific, it's that nearly all points where the derivative is zero are saddle points rather than minima. Note that some portion of this nice behavior seems to be due to design choices in modern architectures, like residual connections, rather than being a general fact about all high dimensional problems. https://arxiv.org/pdf/1712.09913 This paper has some nice visualizations.
Original comment Gradient descent visualization
currymj
Neutral mention · rule
As others have said, you don't actually want the global optimum of a neural network because that would be terrible overfitting. There is some evidence that architectural tricks (like ResNet) that empirically help performance are making the loss landscape "more convex", though. https://arxiv.org/pdf/1712.09913.pdf
Original comment An Idiot’s guide to Support vector machines (2003) [pdf]
hadsed
Neutral mention · rule
I find your comment a little strange. First you say something interesting about Deep Dream being an early attempt at better understanding of vision networks but that it's not enough. Then you say the end is in sight? If you just take the visualization part we have a massive amount of work to do. Understanding how these networks solve problem in sophisticated ways will motivate new advances to mitigate shortcomings. It is a software problem that can have huge benefits. For example, see the paper Visualizing Loss Functions For Neural Networks as an example of the incredible insight to be gained with better visualization. The intuition of loss functions hasn't changed much…
Read the full comment
in the last several decades. If you have a local optimizer it needs to be able to find its way down further than it already is. Anything you can do to remove junk configurations from even being considered during optimization will help. But knowing when that is happening has been really hard because it's just so computationally intensive to generate plots like in that paper. https://arxiv.org/abs/1712.09913
Original comment Greedy, Brittle, Opaque, and Shallow: The Downsides to Deep Learning

Keep following the thread.

More in Machine learning

More from the same topics.