Jaehoon Lee, Yasaman Bahri, Roman Novak
Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
2017 · arXiv
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
7 selected excerpts · 7 total mentions
You may enjoy some works on the connections between gaussian processes, neural networks, and linear(in rkhs) models. https://papers.nips.cc/paper_files/paper/2019/hash/39d929972... https://papers.nips.cc/paper_files/paper/1996/hash/ae5e3ce40... https://arxiv.org/abs/1711.00165
Thanks for the submission. I keep an eye out for connections to potentially cheaper and simpler methods. The new comparison reminds me of this old paper which compared them to Gaussian networks: https://arxiv.org/abs/1711.00165 For causality, I'm also keeping an eye out for ways to tie DNN research back into decision trees, esp probabilistic and nested. Or fuzzy logic. Here's an example I saw moving probabilistic methods into decision trees: http://www.gatsby.ucl.ac.uk/~balaji/balaji-phd-thesis.pdf Many of these sub-fields develop separately in their own ways. Gotta wonder what recent innovations in one could be ported to another.
> How well does a classic deep net architecture like AlexNet or VGG19 classify on a standard dataset such as CIFAR-10 when its “width”— namely, number of channels in convolutional layers, and number of nodes in fully-connected internal layers — is allowed to increase to infinity? Such questions have come to the forefront in the quest to theoretically understand deep learning and its mysteries about optimization and generalization. They also connect deep learning to notions such as Gaussian processes and kernels. A recent paper [Jacot et al., 2018] introduced the Neural Tangent Kernel (NTK) which captures the behavior of fully-connected deep nets in the infinite width…
limit trained by gradient descent; this object was implicit in some other recent papers. An attraction of such ideas is that a pure kernel-based method is used to capture the power of a fully-trained deep net of infinite width. https://openreview.net/pdf?id=rkl4aESeUH, https://github.com/google/neural-tangents > It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. https://arxiv.org/abs/1711.00165 And of course, one needs to look back at SVMs applying a kernel function and separating with a line, which looks a lot like an ANN with a single hidden layer followed by a linear mapping. https://stats.stackexchange.com/questions/238635/kernel-meth...
I believe it's this: https://arxiv.org/abs/1711.00165 And then the NTK paper: https://arxiv.org/abs/1806.07572 There is also this which seems to have been produced in parallel, closely related to the first but not exactly the same: https://arxiv.org/abs/1804.11271
You can do basic math to a multivariate normal distribution to approximate other distributions. In classic feedforward networks, that's achieved by using nonlinear activation functions. Very wide networks turn into Gaussian processes in the limit [0]. There are also approaches explicitly embracing the "basic math on multivariate normal distributions" framework using normalizing flows [1]. [0] https://arxiv.org/abs/1711.00165 [1] https://arxiv.org/abs/1505.05770
There are some interesting connections between GPs and neural networks. Deep neural networks with random weights concentrate about gaussian processes, so the two may not be so far apart in practice. https://arxiv.org/pdf/1711.00165.pdf
How exactly did theoretical equivalence result kill SOM's as field of study? There is similar connection between deep fully connected neural networks and Gaussian process. Deep Neural Networks as Gaussian Processes https://arxiv.org/abs/1711.00165
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Scott Aaronson
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.