Skip to content
The methodOpen data
Back to the collection

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

2015 · arXiv

Machine learningComputer visionPsychology
Read the paper
HN score
0
Mentions
4
Readers
4
Months
4
What’s in a score?

From the abstract

Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66%). To our knowledge, our result is the first to surpass human-level performance (5.1%, Russakovsky et al.) on this visual recognition challenge.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

peterfirefly
Neutral mention · rule
Not necessarily. It turns out to really matter how the weights are initialized (randomly, but from which distribution?) for ordinary feed-forward neural nets that only uses standard neurons -- different activation functions have different optimal initialization distributions. https://machinelearningmastery.com/weight-initialization-for... A big breakthrough was Kaiming He's work on initialization for ReLU: https://arxiv.org/abs/1502.01852 (Kaiming is the personal name, He is the family name. He Kaiming is the native Chinese order. Lots of citations use "Kaiming" by mistake.)
Original comment Direct initialization of transformers using larger pretrained ones
kahnjw
Neutral mention · rule
Words matter, concretely define "open ended"? Did you just add that phrase to preemptively nullify evidence to the contrary? Deep learning has surpassed human level performance on many tasks [1][2]... (could add more you get the point). [1] https://www.sciencedirect.com/science/article/pii/S2215017X1... [2] https://arxiv.org/pdf/1502.01852v1.pdf
Original comment Explainable Deep Learning: A Field Guide for the Uninitiated
tanilama
Neutral mention · rule
https://arxiv.org/abs/1512.03385 Deep Residual Learning for Image Recognition Citation 4000+. Milestone of image recognition. https://arxiv.org/abs/1502.01852 Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Also known as PreLU, citation 1600+. The initialization scheme is widely adapted by a lot of following works. Those are the most influential ones, all come from Kaiming He. He is now in FAIR, and keeps very good publication record, with works like MASK-RCNN, but that doesn't really count as purely Chinese contribution, so we skipped here. Some recent memory: 1. https://arxiv.org/abs/1703.06211 2. https://arxiv.org/abs/1709.01507 3. https://arxiv.org/abs/1707.01083 4. https://arxiv.org/pdf/1608.06993 5. https://arxiv.org/abs/1705.01088 Those are all good papers.
Original comment China is catching up to the USA, while Japan is being left behind
lars
Neutral mention · rule
Since this paper along with [1] both beat human level performance on ImageNet, using seemingly unrelated techniques, it seems like there's a clear path to achieve the next state of the art result: Implement and train a network that uses both Batch Normalization and PReLU's at the same time. Right? 1: http://arxiv.org/pdf/1502.01852v1.pdf
Original comment Batch Normalization: Accelerating Deep Network Training [pdf]

Keep following the thread.

More in Machine learning

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score