Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
2015 · arXiv
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
18 selected excerpts · 18 total mentions
Conceptually, if you’re trying to classify artwork by a category like artist or period, the problem is just ordinary image classification and won’t require anything unique so I’d recommend just reading the current SOA papers for that task. The foundation for many of the current SOA papers is He, et. al: https://arxiv.org/abs/1512.03385 (I wrote and maintain Keras-ResNet and I’m always looking for interesting examples. Let me know if you cook up something interesting!)
No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/ ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385 Multi-Scale Context Aggregation by Dilated Convolutions - https://arxiv.org/abs/1511.07122 Identity Mappings in Deep Residual Networks -…
https://arxiv.org/abs/1603.05027 Recurrent Neural Network Regularization - https://arxiv.org/abs/1409.2329 Deep Speech 2: End-to-End Speech Recognition in English and Mandarin - https://arxiv.org/abs/1512.02595 Order Matters: Sequence to Sequence for Sets - https://arxiv.org/abs/1511.06391 Neural Machine Translation by Jointly Learning to Align and Translate - https://arxiv.org/abs/1409.0473 Pointer Networks - https://arxiv.org/abs/1506.03134 Attention Is All You Need - https://arxiv.org/abs/1706.03762 The Annotated Transformer - https://nlp.seas.harvard.edu/annotated-transformer/ Neural Turing Machines - https://arxiv.org/abs/1410.5401 A Simple Neural Network Module for Relational Reasoning - https://arxiv.org/abs/1706.01427 Relational Recurrent Neural Networks - https://arxiv.org/abs/1806.01822 Neural Message Passing for Quantum Chemistry - https://arxiv.org/abs/1704.01212 Scaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361 GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism - https://arxiv.org/abs/1811.06965 Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077 The First Law of Complexodynamics - https://scottaaronson.blog/?p=762 Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903 Kolmogorov Complexity - https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X Variational Lossy Autoencoder - https://arxiv.org/abs/1611.02731 Machine Super Intelligence - https://www.vetta.org/documents/Machine_Super_Intelligence.p...
> What contributed more towards success in my opinion are "shortcut connections" through layers which enable more influence on early layers during learning. For those who don't know, that is the idea behind ResNet (He et al., Deep Residual Learning for Image Recognition, https://arxiv.org/abs/1512.03385), one of the most influential papers in deep learning of all time. Residual connections make it possible to train networks that are arbitrarily deep. Before ResNet, networks that were too deep were essentially not trainable due to vanishing or exploding gradients.
> The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. Has it ever? I've read a lot of "old" papers and idk if there's ever been a strong theoretical framework. I'd say it is strongest today but not in main papers and those papers are getting rejected because of lack of experiments (wtf is going on with reviewing and why are our solutions "authors have to show reviewers used LLMs"[-1] not "reviewers have to provide meaningful review"? What a joke!). Here's a more recent Universal Approximation paper[0]. Cyberko paper[1] is good, but I think we're moving in the right direction, at…
least with respect to these two works. Other UA papers I've seen are pretty handwavy. But simple also doesn't mean "not useful" or "not worthy of publish" and I think this is something people are forgetting lately. Probably because we're asked to review too much and annoyed, but don't take it out on authors. Some examples are GELU[2] (read the end, there's drama in this paper), Clean-FID (CVPR 22)[3], ViT (ICLR 21) and DeiT (ICML 21)[4,5], and even ResNet (CVPR 16)[6] and tons of ReLU papers we can list. But let's also not confuse big compute with "good work" (they correlate but aren't the same. You can definitely do more with more compute). > he problem with this for me is that we fail to build a nice, crisp understanding of the effects of each design decision in the final outcomes, which hurts the actual “science” of it. It also opens up the field for bogus and unreproducible claims. This is mostly an unsolvable problem in science and I think we actually have bigger problems. This can be solved without conferences due to our community being fairly good about open sourcing code and checkpoints (at least compared to any other research area), as well as just the papers themselves. Reproducibility is only able to be done when people can actually replicate works (lol sorry GPT papers...). What helps this is moving away from publish or perish or allowing people to publish works that are replication attempts (replication is the foundation of science. Why does everything have to be "novel", whatever that means). The bigger problem I see is actually related to the above. Publishing is just insane. It is easy, especially in fast moving times, to claim any work is incremental or not novel. That it's "just x but y". My feelings are "so what?" ViT is "just a transformer on images but image patches are tokens," yet it's insanely useful. It's obvious post hoc but not a priori. If it was, it would already be in use (ViT actually being a good example given the timeline). Everything is simpler and much more obvious after you've already seen it, which is why the whole notion of novelty is ridiculous. We humans just rewrite history in our minds and we also only concentrate on what's popular (lots of incentives, especially with reviewing insanity). Poor Ross Wightman gets shadowed for replications and improving ResNets though (and poor ConvNext). But we're seeing big labs do this and get through because they can do lots of experiments and build a lot of empirical evidence towards work, despite no additional theory. The problem? What about the academics? I don't know how people publish without connections to big labs (aka big compute) anymore. The ideas from the groups are the same (industry is a bit more hyped), but I've seen papers where the theory is good, the architecture makes sense (and is well explained) and my co-reviewers for a fucking workshop want to reject it because "not enough experiments" and "only one 'real world' experiment." These are papers that are on par or better than many I see in the main conferences (you can tell they're shifted to workshops due to rejects). Talk about holding back science. Holding back science is making authors submit works over and over and over to get orthogonal reviews every time. Holding back science is me arguing with my co-reviewers who admitted to not understanding the work that "more experiments" is not a sufficient reason to reject from a workshop (or a main conference). Hell, I got rejected once for a distillation paper and the reviewers complained that my teacher model's performance didn't match that of the original paper (no one has replicated that work btw, and I beat the second best published result I could find). Another paper I got hard rejected because I redacted a github link and an appendix citation broke when splitting the work. Guess what the ACs did? They let that person write a stronger response and then because they were the only high confidence reviewer on that paper (others were 3/5 confidence and borderline/WA) and so rejected the work. What kind of insane system is that? I see instances like this all over the place from many different peers across many different universities. There's more that's broken before we can even begin to talk about other issues because our system of "what qualifies as meaningful work" shouldn't be a fucking slot machine run by undergrads. CVPR has well over 10k submissions. Good luck to the <500 ACs and all the reviewers. God have mercy on the grad students who are just trying to graduate. I'm sure you're all doing good work even if the reviewer didn't read a word. I hope your pictures and tables are enough. [-1] https://twitter.com/CVPR/status/1722384482261508498 [0] https://www.jmlr.org/papers/v23/20-1433.html [1] https://web.njit.edu/~usman/courses/cs675_fall18/10.1.1.441.... [2] https://arxiv.org/abs/1606.08415 [3] https://www.cs.cmu.edu/~clean-fid/ [4] https://arxiv.org/abs/2010.11929 [5] https://arxiv.org/abs/2012.12877 [6] https://arxiv.org/abs/1512.03385
Haven't read thoroughly but it seems they are investigating ResNet models [0] trained for image classification. > We apply CKA to a family of ResNets of varying depths and widths, trained on common benchmark datasets (CIFAR-10, CIFAR-100 and ImageNet) [0] https://arxiv.org/abs/1512.03385
1) How similar is this to creating a resnet? [1]. What are some key differences and similarities? 2) Has a CNN version of this been implemented in PyTorch? [1] https://arxiv.org/pdf/1512.03385.pdf (Figure 2)
Intelligent trial and error. 1. Transformers are an extension of the attention mechanism, a well known LSTM extension that worked well for machine translation (2014, https://arxiv.org/abs/1409.0473). A transformer model is essentially building a multi headed attention module, analogue to CNNs, then stacking several layers of them, analogue to CNNs / stacked LSTMs. (1998, http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf) 2. Transfomers use residual blocks, which were introduced by the ResNet CNN arhitecture (2015, https://arxiv.org/abs/1512.03385). At the time, ResNet was topping the ImageNet benchmarks. This technique helps preventing the vanishing gradient problem…
during training. 3. Transformers use normalization extensively. Layer normalization and attention normalization. This helps keeping internal vectors in the neighborhood of 1 and prevents vanishing gradient training collapse. 4. Correct initialization of the network vectors also helps preventing the vanishing gradients problem. 5. Unsupervised pretraining was one of the first tricks to make deep networks work (2009, https://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf) 6. Pretraining was used extensively in the vision community for transfer learning, i.e. reusing the weights of a network trained on ImageNet and replace the top layers / loss function to tackle a different problem. 7. Finally, language modelling, that is predicting the next word in a sentence, was a well known technique to make machine translation work better. Researchers were looking for better language modelling techniques using large corpora (2013, https://arxiv.org/abs/1312.3005, https://www.kaggle.com/c/billion-word-imputation)
ResNet https://arxiv.org/abs/1512.03385 came out of Microsoft Research Asia. I think that's a pretty good thing they produced. Evidently they managed to hire the top talent behind that work at some point. Looks like Kaiming He works at Facebook now, though. http://kaiminghe.com/
https://arxiv.org/abs/1512.03385 Deep Residual Learning for Image Recognition Citation 4000+. Milestone of image recognition. https://arxiv.org/abs/1502.01852 Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Also known as PreLU, citation 1600+. The initialization scheme is widely adapted by a lot of following works. Those are the most influential ones, all come from Kaiming He. He is now in FAIR, and keeps very good publication record, with works like MASK-RCNN, but that doesn't really count as purely Chinese contribution, so we skipped here. Some recent memory: 1. https://arxiv.org/abs/1703.06211 2. https://arxiv.org/abs/1709.01507 3. https://arxiv.org/abs/1707.01083 4. https://arxiv.org/pdf/1608.06993 5. https://arxiv.org/abs/1705.01088 Those are all good papers.
In natural language processing tasks you see a lot of non-CNN architectures. These usually are designed to be able to deal with sequential data, so some kind of "memory" is needed. Sometimes you see this combined with a CNN. There has been a few question answering systems that have one or more CNN layers. In don't entirely understand these designs, but presumably the convultional layers are an attempt to understand the different orders of words. There are lots of techniques that people use to try to make deep networks work well. Mostly theses are about making errors backprog better. One of the most successful recent innovations is the ResNet architectures (https://arxiv.org/abs/1512.03385), and the related highway networks.
Deep Residual Learning for Image Recognition https://arxiv.org/abs/1512.03385 Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper…
than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. And some good answers here: https://www.quora.com/How-does-deep-residual-learning-work
I'm not qualified to answer this, but I will anyway. To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there are higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5] that are made up of simpler blocks. Learning what these do and when it's appropriate to use them requires following current literature. Operating neural networks requires some systems engineering skill. It takes a long time to train a single network and you'll…
find yourself trying many different architectures and hyperparameters along the way. Because of this, you'll want to distribute the training across many different systems and be able to easily monitor and deploy jobs on those systems. A solid grasp of mathematics is useful to effectively debug your networks. You'll frequently find your network doesn't converge or gives totally garbage results, so you need to know how to dig into the network internals and understand how everything works. This is especially true if you're implementing a new building block from a paper. Finally, know your machine learning and statistics fundamentals. Understand overfitting, model capacity, cross validation, probability, model ensembles, information theory, and so on. Know when a simpler model is more appropriate. [1] ftp://ftp.idsia.ch/pub/juergen/fki-207-95.ps.gz [2] http://arxiv.org/abs/1409.1259 [3] http://arxiv.org/abs/1505.00387 [4] http://arxiv.org/abs/1502.03167 [5] http://arxiv.org/abs/1512.03385
Having not read the paper, something I find unclear: is it only the feedback path that is skipped, or is the feedforward path also skipped? The abstract mentions replacing the layer with an identity function. I'm not sure how this would work, wouldn't it change the result (i.e the encoding used by the following layer would be corrupted) if you just multiply the inputs by 1 and add them? Otherwise, how precisely do you "skip" a layer without corrupting the training of lower layers? Edit: the answer is in the definition of "skip layers", introduced in a previous paper: http://arxiv.org/abs/1512.03385 which introduces identity functions into the layer equation.. I guess I have more reading to do on this topic.
Yes. The general task of looking at arbitrary images and labeling objects (from a set of known categories) in those images is called "detection." In fact the problem you described is easier, because there's only one category (people), and the system only needs to provide a count, rather than provide bounding box rectangles around each object (which is what the standard "detection" task entails). Convolutional neural nets are the state of the art for this, specifically deep residual learning (http://arxiv.org/abs/1512.03385). It requires a good deal of background to understand what's going on and tune/implement the models, though, even if you just use the frameworks…
already out there. You probably don't even need that much data - you can probably grab pre-trained models and train them on a small additional dataset you collect. They can definitely handle arbitrary backgrounds, although having a standard background makes the problem even easier, again. Most deep learning computer vision algos are trained on 256x256 images, so having even larger images is just fine (you can downsample, or maybe even add up the predictions of different crops).
I've been trying to read a paper a day since midsummer. These are a few of the interesting, for me personally, since then: Generating Sequences With Recurrent Neural Networks - http://arxiv.org/abs/1308.0850 Older one, but important to understand deeply since other recent ideas have come from this! Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks - http://arxiv.org/abs/1511.06434 Unitary Evolution Recurrent Neural Networks - http://arxiv.org/abs/1511.06464 State of the Art Control of Atari Games Using Shallow Reinforcement Learning - http://arxiv.org/abs/1512.01563 Interesting discussion in section 6.1 on the…
shortcomings/issues of DQN done by Deepmind Spectral Representations for Convolutional Neural Networks - http://arxiv.org/abs/1506.03767 Deep Residual Learning for Image Recognition - http://arxiv.org/abs/1512.03385 Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) - http://arxiv.org/abs/1511.07289 I wish they did more comparisons between similar network architecture with only the units swapped out. Eg. Alexnet, Relu vs Alexnet, Elu. On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models - http://arxiv.org/abs/1511.09249 Just a few from my list :)
Deep Residual Learning for Image Recognition http://arxiv.org/abs/1512.03385 Batch Normalization http://jmlr.org/proceedings/papers/v37/ioffe15.pdf Deep Neural Decision Forests http://research.microsoft.com/pubs/255952/ICCV15_DeepNDF_mai... Spatial Transformer Networks https://papers.nips.cc/paper/5854-spatial-transformer-networ...
It's mentioned in the article, but here's a direct link to the paper on arXiv: http://arxiv.org/abs/1512.03385 And direct link to the PDF: http://arxiv.org/pdf/1512.03385v1.pdf
For more context to people, we have the universal approximation theorem for neural nets that basically says if a network is wide enough it can approximate anything (with at least 2 layers). So a lot of stuff was really wide. Then VGG[0] came out and showed that deep networks were very effective (along with other papers, things happen in unison. Leibniz and Newton). Then you get ResNets[1] with skip connections and move forward to today. Today we've started looking more at what networks are doing and where their biases lie. This is because we're running into some roadblocks with CNNs vs Transformers. They have different inductive biases. Vision transformers still aren't…
defeating CNNs, but they are close and it is clear they learn different things. So we're seeing more papers doing these types of analyses. ML will likely never be fully interpretable, but we're getting better at understanding. This is good because a lot of times picking your model and network architecture is more art than science (especially when choosing hyper parameters). [0] https://arxiv.org/abs/1409.1556 [1] https://arxiv.org/abs/1512.03385
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.