Skip to content
The methodOpen data
Back to the collection

Neural Ordinary Differential Equations

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud

2018 · arXiv

Machine learning
Read the paper
HN score
0
Mentions
10
Readers
10
Months
7
What’s in a score?

From the abstract

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

10 selected excerpts · 10 total mentions

Scene_Cast2
Neutral mention · rule
People have mentioned the discrete - continuous tradeoff. One way to bridge that gap would be to use https://arxiv.org/abs/1806.07366 - they draw an equivalence between vanilla (FC layer) neural nets of constant width with differential equations, and then use a differential equation solver to "train" a "neural net" (from what I remember - it's been years since that paper...). Another approach might be to take an information theoretic view with the infinite-width finite-entropy nets.
Original comment Where is Noether's principle in machine learning?
rubatuga
Neutral mention · rule
Yeah. This kind of mirrors my thought that our visual cortex is excellent at split-second image recognition, for things like identifying predators, dangers, etc, and this lack of "thinking" is why our current generation of neural networks has matched the performance of humans on image recognition tasks. I agree that there is a "time" component necessary for improved thinking or intelligence, and since our current neural networks don't have these dynamics, they are lacking. Extremely deep neural networks, especially with residual connections, IMO are better approximations of "thinking". Models such as ALBERT demonstrate that duplication of the same layer can still perform…
Read the full comment
extremely well on NLP tasks. I'm still intrigued by the Neural Ordinary Differential Equations paper [0] and the research in that direction. I also remember reading about differentiable neural computers, but I haven't followed the progress of that at all. 0: https://arxiv.org/abs/1806.07366
Original comment Retrospective review of Gödel, Escher, Bach (1996) [pdf]
damip
Neutral mention · rule
This work focuses on steady-state computing, but it could also be interesting to use transient physical behaviors to process time-varying signals Maybe by modelling dynamical systems as "neural nets" as in: https://arxiv.org/abs/1806.07366 and https://arxiv.org/abs/1808.08412 Or by using complicated physical systems we don't even understand to build Echo State Networks: http://www.scholarpedia.org/article/Echo_state_network
Original comment ‘A Swiss cheese-like material’ that can solve equations
cs702
Neutral mention · rule
Nice work. I can think of only two things that are missing: * Normalizing flows - e.g., https://arxiv.org/abs/1605.08803 , https://arxiv.org/abs/1807.03039 , among many others * ODEnets and continous normalizing flows - https://arxiv.org/abs/1806.07366
Original comment Deep Learning State of the Art (2019) [video]
joshuamorton
Neutral mention · rule
>We've got a winner. Tensors "do" something; they're functions. And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video). >neural networks as ODEs You're a few months late :P [0] [0]: https://arxiv.org/abs/1806.07366
Original comment Tensor Considered Harmful Part 2
ArtWomb
Neutral mention · rule
Neural ODE solver just announced as a Best Paper at NeurIPS 2018 conference. What's interesting is that not only do preliminary results yield significantly less error in prediction vs RNNs. Differentiable models allow for extrapolations well beyond range of observable test data set. Neural Ordinary Differential Equations https://arxiv.org/pdf/1806.07366.pdf
Original comment Building a Language and Compiler for Machine Learning

Keep following the thread.

More in Machine learning

More from the same topics.