Brendan Fong, David I. Spivak, Rémy Tuyéras
2017 · arXiv
A supervised learning algorithm searches over a set of functions $A \to B$ parametrised by a space $P$ to find the best approximation to some ideal function $f\colon A \to B$. It does this by taking examples $(a,f(a)) \in A\times B$, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent---with respect to a fixed step size and an error function satisfying a certain property---defines a monoidal functor from a category of parametrised functions to this category of update rules. This provides a structural perspective on backpropagation, as well as a broad generalisation of neural networks.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
In fact, the compositional structure underlying that of predictive coding [0,1] is abstractly the same as that underlying backprop [2]. (Disclaimer: [0,1] are my own papers; I'm working on a more precise and extensive version of [1] right now!) [0] https://arxiv.org/abs/2006.01631 [1] https://arxiv.org/abs/2101.10483 [2] https://arxiv.org/abs/1711.10455
You may be interested in this paper too https://arxiv.org/abs/1711.10455 I think given a language with AD you already don't need to have a separate language for graph building and my understanding is that's one of the goals for the Julia team and I've seen other nice looking packages for AD in other languages * https://fluxml.ai/2019/02/07/what-is-differentiable-programm... * http://diffsharp.github.io/DiffSharp/examples-inversekinemat... I'm also very excited about being able to design neural nets using ideas from functional programming. I love the Keras API but I don't think it's going to be the obvious API a couple of years down the line.
David Spivak has done lots of cool work on applying category theory to many concrete systems like databases [1]. Brendan Fong has worked on applying category theory to things that act like interconnected networks, such as electrical circuits. This fits into a larger ambitious program that John Baez is part of to use category theory to understand physical systems that are still much too complicated to understand with current mathematical tools, such as ecosystems. Fong's thesis is on my reading list [2]. The two authors are doing great research, like this investigation of the algebra of backpropagation [3]. I really look forward to reading this too! [1] https://arxiv.org/abs/0904.2012 [2] https://arxiv.org/abs/1609.05382 [3] https://arxiv.org/abs/1711.10455
After reading "Backprop as Functor: A compositional perspective on supervised learning", it came to my attention that the spaCy backend (thinc) is built with higher order functions instead of a computational graph (unlike tensorflow, chainer, or pytorch) https://github.com/explosion/thinc#no-computational-graph--j... https://arxiv.org/abs/1711.10455 Could someone give me some more detail?
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.