Skip to content
The methodOpen data
Back to the collection

Universal Differential Equations for Scientific Machine Learning

Christopher Rackauckas, Yingbo Ma, Julius Martensen

Show all 9 authors

Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, Alan Edelman

2020 · arXiv

Science & researchMachine learningStatistics & probability
Read the paper
HN score
0
Mentions
8
Readers
3
Months
6
What’s in a score?

From the abstract

In the context of science, the well-known adage "a picture is worth a thousand words" might well be "a model is worth a thousand datasets." In this manuscript we introduce the SciML software ecosystem as a tool for mixing the information of physical laws and scientific models with data-driven machine learning approaches. We describe a mathematical object, which we denote universal differential equations (UDEs), as the unifying framework connecting the ecosystem. We show how a wide variety of applications, from automatically discovering biological mechanisms to solving high-dimensional Hamilton-Jacobi-Bellman equations, can be phrased and efficiently handled through the UDE formalism and its tooling. We demonstrate the generality of the software tooling to handle stochasticity, delays, and implicit constraints. This funnels the wide variety of SciML applications into a core set of training mechanisms which are highly optimized, stabilized for stiff equations, and compatible with distributed parallelism and GPU accelerators.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

8 selected excerpts · 8 total mentions

ChrisRackauckas
Neutral mention · rule
Yeah, Julia does great in this domain. DataDrivenDiffEq.jl (https://datadriven.sciml.ai/) is a comprehensive symbolic regression package out there, which pulls together all of what's mentioned in the article (SINDy and variants, EQSearch, SymbolicRegression.jl (which is the core of PySR), OccamNet, etc.) all into a single API to allow comparisons between each of the methods. The Julia automated discovery of physical equations work showcases these symbolic regression techniques quite heavily, for example in the State of SciML talk (https://www.youtube.com/watch?v=eSeY4K4bITI) we discuss how this has been used now across hundreds of different scientific use cases. The…
Read the full comment
universal differential equations paper which describes the Julia SciML organization (https://arxiv.org/abs/2001.04385) demonstrates these cases where neural networks in differential equations are mixed with symbolic regression to allow for mixing prior physical knowledge with data for discovering just the unknown higher order physical equations. And there's a lot more directions we're going next.
Original comment ‘Machine Scientists’ Distill the Laws of Physics from Raw Data
Libbum
Neutral mention · rule
To respond to both the parent question, and this comment: indeed, this is black-box optimal control in essence. However, this method is just one small aspect of the SciML [0] ecosystem now. The article is a little outdated in that sense. Once obtaining your NN control parameter, it's now possible to use Sparse Identification of Nonlinear Dynamics (SINDy) on that parameter to recover equations of motion governing it [1]. The real promise of these methods is to use the universal approximator power of NNs to get around the 'curse of dimensionality' & uncover presently unknown representations of motion within any system. Take a look at [2] for a more detailed description. [0]: https://sciml.ai/ [1]: https://datadriven.sciml.ai/dev/sparse_identification/sindy/ [2]: https://arxiv.org/abs/2001.04385
Original comment Differentiable Control Problems
ChrisRackauckas
Neutral mention · rule
It's a good question, which I answer in much more depth here: https://www.reddit.com/r/MachineLearning/comments/ryw53x/d_f... . Specifically that answer is about NNs vs fourier series, but it's the same point: polynomials, Fourier series, and NNs are all universal function approximators, so why use a NN? If you write out what happens though, you can easily see the curse of dimensionality in action and see why a 100-dimensional function shouldn't be approximated by tensor products of polynomials. See the rest there. So it's not really about NNs vs polynomials/fourier/Chebyshev, etc., but rather what is the right thing to use at a given time. In the universal differential…
Read the full comment
equations paper (https://arxiv.org/abs/2001.04385), we demonstrate that some examples of automatic equation discovery are faster using Fourier series than NNs (specifically there the discovery of a semilinear partial differential equation). That doesn't say that NNs are a bad way to do all of this, it just means that they are one trainable object that is good for high dimensional approximations, but libraries should allow you to easily move between classical basis functions and NNs to best achieve the highest performance.
Original comment Differentiable Programming – A Simple Introduction
ChrisRackauckas
Neutral mention · rule
Yes, and our recent work https://arxiv.org/abs/2001.04385 gives a fairly general form for how to mix known scientific structural knowledge directly with machine learning. In fact, some of these PDE solvers are just instantiations of specific choices of universal differential equations. I agree that in many cases the "fully uninformed" physics-informed neural network won't work well, but we need to fully optimize a library with all of the training techniques possible in order to prove that, which is what we plan to do. In the end, I think PINNs will be most applicable to (1) non-local PDEs where classical methods have not fared well, so things like fractional differential…
Read the full comment
equations, and (2) very high dimensional PDEs, like 100's of dimensions, but paired with constraints on the architecture to preserve physical quantities and relationships. But of course, something like a fractional differential equation is not an example for the first pages of tutorials since they are quite niche equations to solve!
Original comment AI in physics: are we facing a scientific revolution?
ChrisRackauckas
Neutral mention · rule
A lot of these methods here utilize the universal differential equation framework described here: https://arxiv.org/abs/2001.04385 . Specifically, the last example in this preprint describes how high dimensional parabolic PDEs can be solving using neural networks inside of a specific SDE (derivation in the supplemental). Discrete physics-informed neural networks also are a subset of this methodology. The other subset of methods, continuous physics-informed neural networks, are described in https://www.sciencedirect.com/science/article/pii/S002199911... . For a very basic introduction, I wrote some lecture notes on how this is done for a simple ODE with code examples: https://mitmath.github.io/18S096SciML/lecture2/ml
Original comment AI in physics: are we facing a scientific revolution?
ChrisRackauckas
Neutral mention · rule
Scientific machine learning. The ability to mix scientific knowledge with machine learning in order to allow it to extrapolate accurately from small amounts of data. As machine learning tries to move towards areas where large-scale data gathering is prohibitively expensive or even unethical (like in clinical trials: you can't have a million failures to train a neural network!), mixing scientific models with machine learning will be required to bring AI into these domains in a way that's beyond hype. Physics-informed neural networks [1], differentiable programming [2], universal differential equations [3], SInDy [4], etc. are all ideas directions I think is the next direction for machine learning. [1] https://www.sciencedirect.com/science/article/pii/S0021999118307125 [2] https://arxiv.org/abs/1907.07587 [3] https://arxiv.org/abs/2001.04385 [4] https://www.pnas.org/content/113/15/3932
Original comment Ask HN: What’s a big trend we should all be following?
ChrisRackauckas
Neutral mention · rule
The Julia Lab (MIT) | DevOps Engineer (+ More) | Onsite or Remote | Full Time | https://julia.mit.edu/ The Julia Lab is a lab in MIT's CSAIL which focuses on the development of software tooling for solving large-scale scientific problems. Named after it's original result, the Julia programming language, the Julia Lab's research ranges from scientific machine learning (SciML) [1] and differentiable programming [2] to large-scale heterogeneous parallel computing [3]. The lab is currently growing in response to new directions and collaborations, and we are looking for both researchers and devops engineers to help with the research activities. We are looking for individuals…
Read the full comment
who can help accelerate scientific machine learning workflows with gitops, develop high-precision continuous benchmarking systems, build and wrap essential scientific computing binaries into Julia packages, and help with the deployment of Julia-based software to alternative architectures. Or those with sufficient compiler, numerical computing, and scientific machine learning knowledge may find alternative projects. Experience with Julia is highly preferred. [1] Universal Differential Equations for Scientific Machine Learning, https://arxiv.org/abs/2001.04385 [2] A Differentiable Programming System to Bridge Machine Learning and Scientific Computing, https://arxiv.org/abs/1907.07587 [3] Rapid software prototyping for heterogeneous and distributed platforms, https://www.sciencedirect.com/science/article/abs/pii/S09659...
Original comment Ask HN: Who is hiring right now?

Keep following the thread.

More in Science & research

More from the same topics.