Skip to content
The methodOpen data
Back to the collection

Highway Networks

Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber

2015 · arXiv

Machine learningStatistics & probability
Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

There is plenty of theoretical and empirical evidence that depth of neural networks is a crucial ingredient for their success. However, network training becomes more difficult with increasing depth and training of very deep networks remains an open problem. In this extended abstract, we introduce a new architecture designed to ease gradient-based training of very deep networks. We refer to networks with this architecture as highway networks, since they allow unimpeded information flow across several layers on "information highways". The architecture is characterized by the use of gating units which learn to regulate the flow of information through a network. Highway networks with hundreds of layers can be trained directly using stochastic gradient descent and with a variety of activation functions, opening up the possibility of studying extremely deep and efficient architectures.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

walrus
Neutral mention · rule
I'm not qualified to answer this, but I will anyway. To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there are higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5] that are made up of simpler blocks. Learning what these do and when it's appropriate to use them requires following current literature. Operating neural networks requires some systems engineering skill. It takes a long time to train a single network and you'll…
Read the full comment
find yourself trying many different architectures and hyperparameters along the way. Because of this, you'll want to distribute the training across many different systems and be able to easily monitor and deploy jobs on those systems. A solid grasp of mathematics is useful to effectively debug your networks. You'll frequently find your network doesn't converge or gives totally garbage results, so you need to know how to dig into the network internals and understand how everything works. This is especially true if you're implementing a new building block from a paper. Finally, know your machine learning and statistics fundamentals. Understand overfitting, model capacity, cross validation, probability, model ensembles, information theory, and so on. Know when a simpler model is more appropriate. [1] ftp://ftp.idsia.ch/pub/juergen/fki-207-95.ps.gz [2] http://arxiv.org/abs/1409.1259 [3] http://arxiv.org/abs/1505.00387 [4] http://arxiv.org/abs/1502.03167 [5] http://arxiv.org/abs/1512.03385
Original comment Tinker with a Neural Network in Your Browser
336f5
Neutral mention · rule
http://arxiv.org/abs/1505.00387 Very deep NNs, which try to bypass the training difficulties in propagating error all the way from the bottom to the top by adding some special shortcuts between nodes in distant layers.
Original comment Towards Neural Network-based Reasoning

Keep following the thread.

More in Machine learning

More from the same topics.