Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger
2016 · arXiv
Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections - one between each layer and its subsequent layer - our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet .
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
5 selected excerpts · 5 total mentions
You're correct, I also added some info in a sibling comment that might interest you. But you might also be interested in reading the DenseNet paper[0]. While it's convs, the ideas still apply. There's actually a lot of research in this topic though they'll be in more "theory" papers and the ideas are slow to propagate into the mainstream works. [0] https://arxiv.org/abs/1608.06993
Here is the first (?) application of deep learning to improve the pornography experience, an Android app called Melondream published in early 2017: http://driftwheeler.com Surprisingly, almost 5 years after being published, this app still serves thousands of user sessions per day. That rate continues to slowly grow. Except for its user interface, the whole app (client and server) was written in Go (https://pkg.go.dev/golang.org/x/mobile/cmd/gomobile). It uses densenet feature vectors (https://arxiv.org/abs/1608.06993) to allow "touch-to-search" (ha!) with a fractional norm distance metric (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.23...). Citation: Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks (https://arxiv.org/abs/1406.6909).
https://arxiv.org/abs/1512.03385 Deep Residual Learning for Image Recognition Citation 4000+. Milestone of image recognition. https://arxiv.org/abs/1502.01852 Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Also known as PreLU, citation 1600+. The initialization scheme is widely adapted by a lot of following works. Those are the most influential ones, all come from Kaiming He. He is now in FAIR, and keeps very good publication record, with works like MASK-RCNN, but that doesn't really count as purely Chinese contribution, so we skipped here. Some recent memory: 1. https://arxiv.org/abs/1703.06211 2. https://arxiv.org/abs/1709.01507 3. https://arxiv.org/abs/1707.01083 4. https://arxiv.org/pdf/1608.06993 5. https://arxiv.org/abs/1705.01088 Those are all good papers.
* Densely Connected Convolutional Networks [0] by Gao Huang, Zhuang Liu, Laurens van der Maaten, & Kilian Q. Weinberger. * Learning from Simulated and Unsupervised Images through Adversarial Training [1] by Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, & Russell Webb CVPR 2017 Best Paper Honorable Mention Awards * Annotating Object Instances with a Polygon-RNN [2] by Lluís Castrejón, Kaustav Kundu, Raquel Urtasun, & Sanja Fidler * YOLO9000: Better, Faster, Stronger [3] by Joseph Redmon & Ali Farhadi CVPR 2017 Best Student Paper Award * Computational Imaging on the Electric Grid [4] by Mark Sheinin, Yoav Y. Schechner, & Kiriakos N.…
Kutulakos CVPR 2017 Longuet-Higgins Prize (for cvpr 2007 papers that stood the test of time) * Object Retrieval with Large Vocabularies and Fast Spatial Matching [5] by James Philbin, Ondrej Chum, Michael Isard, Josef Sivic & Andrew Zisserman CVPR 2017 PAMI Young Researcher Award * Ross Girshick & Julien Mairal [0] https://arxiv.org/abs/1608.06993 [1] https://arxiv.org/abs/1612.07828 [2] https://arxiv.org/abs/1704.05548 [3] https://arxiv.org/abs/1612.08242 [4] http://openaccess.thecvf.com/content_cvpr_2017/papers/Sheini... [5] http://ieeexplore.ieee.org/document/4270197/
For deep convnets the vanishing gradient problems can mostly be solved by using residual architectures. See: https://arxiv.org/abs/1603.05027 This is kind of related to solving the vanishing gradient issue in RNNs by using additive recurrent architectures like LSTMs and GRUs. Alternatively it's possible to use concatenative skip connections as in DenseNets: https://arxiv.org/abs/1608.06993 Still using 1000 layers is useless in practice. State of the art image classification models are in the range 30-100 layers with residual connections and varying numbers of channels per layer depending on the depth so as to keep a tractable total number of trainable parameters. The 1000 layers nets are just interesting as a memory scalability benchmark for DL frameworks and to validate empirically the feasibility of the optimization problem but are of no practical use otherwise (as far as I know).
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.