Yoshua Bengio
2012 · arXiv
Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
I would add "Practical recommendations for gradient-based training of deep architectures" to the list for those who already have a feel for training multi-layer neural nets. It provides a good overview for those that want to learn more about gradient descent, hyperparameter tuning, and other practical considerations involved with training deep architectures. http://arxiv.org/abs/1206.5533
To address some of the comments being presented here, neural nets despite being harder to train can be debugged visually. A few tips for those of you who use neural nets: Debug the weights with histograms. Track the gradient and make sure the magnitude is not too large and its normally distributed. Keep track of your gradient changes when using either gradient descent or conjugate gradient. Plot your filters, visualize what each neuron is learning. Watch the rate of change of your cost function. If it seems like its changing too fast and stops early lower your learning rate. Plot your activations: if they start out grey you're fine. If you start all black, you need to…
retune some of your parameters. Lastly, understand the algorithm you're using. Convolutional nets are different from recursive neural tensor are different denoising autoencoders are different from RBMS/DBNs. Pay attention to your cost function, reconstruction entropy is used differently from negative log likelihood is used differently for different objectives. If you are trying to do feature learning, you are using RBMs, Denoising AutoEncoders and you will use reconstruction entropy. This is what you use for feature detectors. You may end up using negative log likelihood if you are dealing with continuous data. For RBMs, pay attention to the different kinds of units[1]. Hinton recommends Gaussian visible with recitifed linear for continuous data, binary binary otherwise. For denoising autoencoders, watch your corruption level. A higher one helps generalize better, especially with less data. For time series or sequential data, you can use a recurrent net,moving window with DBNs, or recursive neural tensor Other knobs: If your deep learning framework doesn't have adagrad find one that does. Dropout: crucial. Dropout is used in combination with mini batch learning to handle learning different "poses" of images as well as generalizing feature learning. This can be used in combination with sampling with replacement to minimize sampling error. Regularization: L2 is typically used. Hinton once said: you want a neural net that always overfits but is regularized (youtube video...don't remember link right now). Would love to answer questions! Source: I work on/teach this stuff. Still working my way up there, but it seems to be going well so far.[2][3] Lastly, tweak one knob at a time. Neural nets have a lot going on. You don't want a situation where you A/B tested 10 different parameters at once and you don't know which one worked or why. [1]: http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf [2]: http://deeplearning4j.org/ [3]: http://zipfianacademy.com/ [4]: http://arxiv.org/abs/1206.5533 http://deeplearning4j.org/ http://deeplearning4j.org/debug.html http://yosinski.com/media/papers/Yosinski2012VisuallyDebuggi...
Check out this paper: Practical recommendations for gradient-based training of deep architectures, Y. Bengio http://arxiv.org/abs/1206.5533 There is a section on weight initialization on page 15. In general, this paper has a lot of good information in one place.
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.