Skip to content
The methodOpen data
Back to the collection

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy

2015 · arXiv

Machine learningComputer vision
Read the paper
HN score
0
Mentions
4
Readers
3
Months
4
What’s in a score?

From the abstract

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

walrus
Neutral mention · rule
I think they're referring to the momentum parameter at [1]. The exponential moving average (EMA) of the batch mean/variance is used in the batch normalizing transform (Algorithm 1 in [2]). The momentum ranges from 0 to 1. If it's close to 1, which the default of 0.99 is, the EMA of the batch mean/variance will change slowly across batches. If it's close to 0, the EMA will be close to the mean/variance of the current batch. The EMA acts as a low-pass filter. With a momentum close to 1, the EMA changes slowly, filtering out high frequencies and leaving only frequencies close to DC. Note that this is opposite to what grandparent says: 0.99 has a lower frequency cutoff than 0.6 does. So I'm not really sure what they're getting at there. [1] https://keras.io/layers/normalization/#batchnormalization [2] https://arxiv.org/abs/1502.03167
Original comment Deep Forest: Towards an Alternative to Deep Neural Networks
sja
Neutral mention · rule
Good list! I think it's important to note that this article is (intentionally) focused on modern CNN architectures, and not "deep learning" in general. I'd also add in the following "technique" articles: Geoff Hinton et al.'s dropout paper[0] and Loffe and Szegedy's Batch Normalization paper[1]. I don't think there's been enough time for the dust to settle, but I'm excited about the possibilities Stochastic Depth[2] could offer, too. [0]: http://arxiv.org/abs/1207.0580 [1]: http://arxiv.org/abs/1502.03167 [2]: http://arxiv.org/abs/1603.09382
Original comment Some of the most important deep learning papers
twelfthnight
Neutral mention · rule
http://image-net.org/challenges/LSVRC/2012/results.html One team at 15% http://www.image-net.org/challenges/LSVRC/2014/results Most teams are below 15%, GooLeNet is at 6% http://www.image-net.org/challenges/LSVRC/2014/results Microsoft is now below 5% http://arxiv.org/pdf/1502.03167.pdf Google is now below 5% I can't really argue whether that's exponential or not, but it amazing progress in a short amount of time.
Original comment The state of Computer Vision and AI: we are really, really far (2012)
walrus
Neutral mention · rule
I'm not qualified to answer this, but I will anyway. To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there are higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5] that are made up of simpler blocks. Learning what these do and when it's appropriate to use them requires following current literature. Operating neural networks requires some systems engineering skill. It takes a long time to train a single network and you'll…
Read the full comment
find yourself trying many different architectures and hyperparameters along the way. Because of this, you'll want to distribute the training across many different systems and be able to easily monitor and deploy jobs on those systems. A solid grasp of mathematics is useful to effectively debug your networks. You'll frequently find your network doesn't converge or gives totally garbage results, so you need to know how to dig into the network internals and understand how everything works. This is especially true if you're implementing a new building block from a paper. Finally, know your machine learning and statistics fundamentals. Understand overfitting, model capacity, cross validation, probability, model ensembles, information theory, and so on. Know when a simpler model is more appropriate. [1] ftp://ftp.idsia.ch/pub/juergen/fki-207-95.ps.gz [2] http://arxiv.org/abs/1409.1259 [3] http://arxiv.org/abs/1505.00387 [4] http://arxiv.org/abs/1502.03167 [5] http://arxiv.org/abs/1512.03385
Original comment Tinker with a Neural Network in Your Browser

Keep following the thread.

More in Machine learning

More from the same topics.