Xi Cheng, Bohdan Khomtchouk, Norman Matloff, Pete Mohanty
2018 · arXiv
Despite the success of neural networks (NNs), there is still a concern among many over their "black box" nature. Why do they work? Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models. This view will have various implications for NNs, e.g. providing an explanation for why convergence problems arise in NNs, and it gives rough guidance on avoiding overfitting. In addition, we use this phenomenon to predict and confirm a multicollinearity property of NNs not previously reported in the literature. Most importantly, given this loose correspondence, one may choose to routinely use polynomial models instead of NNs, thus avoiding some major problems of the latter, such as having to set many tuning parameters and dealing with convergence issues. We present a number of empirical results; in each case, the accuracy of the polynomial approach matches or exceeds that of NN approaches. A many-featured, open-source software package, polyreg, is available.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
There's also work in polynomial regression as an alternative to regular, neural networks. Polynomial Regression As an Alternative to Neural Nets (2018) https://arxiv.org/abs/1806.06850 Π-nets: Deep Polynomial Neural Networks (2020) https://arxiv.org/abs/2003.03828
The central conceit of the paper (https://arxiv.org/abs/1806.06850) is the "NN<->PR correspondence" from section 6, which argues that since nice nonlinear activation functions like tanh can be arbitrarily well-approximated by polynomials, then the whole stack of convolutional layers can be arbitrarily well-approximated by polynomials, and that means the whole network is just a polynomial. And sure, ```technically''' that's true but there are good reasons that numerics aren't just all polynomials all the time (check out Lloyd Trefethen's 'Approximation Theory and Approximation Practice', for example). tanh is nice and smooth but its Taylor series doesn't converge outside a…
small region, and the whole reason you'd use tanh in the first place is because you'll have things outside that region. And tanh is probably the best case for Matloff's argument! ReLU has that corner, and I don't even want to think about approximating softmax with a polynomial. The blog has comments by, among others, Radford Neal and Kenghagho which I find compelling.
That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression. https://arxiv.org/abs/1806.06850
"The real promise of these methods is to use the universal approximator power of NNs...", still if one is to use a grey-box non-linear model dx/dt = F(x, u, t), why use NNs to characterize F? I would be more comfortable using a polynomial to characterize non-linearity than a "deep" black-box. Polynomials are much easier to "train" because it is just one linear regression with no iteration. It has also been hinted that NN are in essence polynomial regressions [0]. Furthermore, most activation functions are base on e^x where the actual implementation of e^x in a computer is again a polynomial! [0] https://arxiv.org/abs/1806.06850
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Scott Aaronson
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.