Mike Innes, Alan Edelman, Keno Fischer
Chris Rackauckas, Elliot Saba, Viral B Shah, Will Tebbutt
2019 · arXiv
Scientific computing is increasingly incorporating the advancements in machine learning and the ability to work with large amounts of data. At the same time, machine learning models are becoming increasingly sophisticated and exhibit many features often seen in scientific computing, stressing the capabilities of machine learning frameworks. Just as the disciplines of scientific computing and machine learning have shared common underlying infrastructure in the form of numerical linear algebra, we now have the opportunity to further share new computational infrastructure, and thus ideas, in the form of Differentiable Programming. We describe Zygote, a Differentiable Programming system that is able to take gradients of general program structures. We implement this system in the Julia programming language. Our system supports almost all language constructs (control flow, recursion, mutation, etc.) and compiles high-performance code without requiring any user intervention or refactoring to stage computations. This enables an expressive programming model for deep learning, but more importantly, it enables us to incorporate a large ecosystem of libraries in our models in a straightforward way. We discuss our approach to automatic differentiation, including its support for advanced techniques such as mixed-mode, complex and checkpointed differentiation, and present several examples of differentiating programs.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
7 selected excerpts · 7 total mentions
Possibly. On this topic (machine learning, differentiable programming, GPU and parallel computing) I'd recommend the following videos: https://youtu.be/FGfx8CQHdQA https://youtu.be/OcUXjk7DFvU https://arxiv.org/abs/1907.07587 https://youtu.be/7Yq1UyncDNc https://youtu.be/_E2zEzNEy-8 https://youtu.be/6ntJ_al4oXA https://youtu.be/HfiRnfKxI64
I think someone has translated this to J, but I am trying on my own to practice my J-fu by implementing it in my own way. Then I usually open it up to the J experts on the mailing list, and my learning takes off. There are some awesomely smart people there who are generous with their time. Yes, the takeaway is that with APL or J is that you can see the mechanics in a paragraph of code, and it is not a very trivial example. If the libraries or verbs are created to deal with some of the speed or efficiency issues, it is promising as a way of understanding the concept better. The dataframes of R and Python (Pandas) were always a thing in APL/J/k/q, so it is their lingua…
franca or basic unit of computation upon which the languages were built - arrays, not a library. More importantly, almost along the lines of the emperor has no clothes, is a tack to get away from the black box, minimal domain knowledge, ML or DL that cannot be explained too easily - see newly proposed "Algorithmic Accountability Act" in US legislature. Differentiable Programming and AD (Automatic Differentiation)applied with domain knowledge to create a more easily explainable model, and try to avoid biases that may creep into a model and affect health care and criminal systems in a negative way [1][2]. And then there are those who use DL/ANNs for everything, even things that are easily applied and solved using standard optimization techniques. Forest from the trees kind of phenomenon. I have been guilty of getting swept up with them too. I started programming ANNs in the late 80s to teach myself about this new, cool-sounding thing called "neural networks" back then ;) [1] https://arxiv.org/abs/1811.10154 [2] https://arxiv.org/abs/1907.07587
It's already been done by another group of researchers.[1] I'm not sure if that was a factor in their choice of Swift. [1] https://arxiv.org/abs/1907.07587
"A Differentiable Programming System to Bridge Machine Learning and Scientific Computing" https://arxiv.org/abs/1907.07587 https://news.ycombinator.com/item?id=20477873 From the abstract: > We describe Zygote, a Differentiable Programming system that is able to take gradients of general program structures. We implement this system in the Julia programming language. Our system supports almost all language constructs (control flow, recursion, mutation, etc.) and compiles high-performance code without requiring any user intervention or refactoring to stage computations. Just linking to this for those who haven't seen it.
Scientific machine learning. The ability to mix scientific knowledge with machine learning in order to allow it to extrapolate accurately from small amounts of data. As machine learning tries to move towards areas where large-scale data gathering is prohibitively expensive or even unethical (like in clinical trials: you can't have a million failures to train a neural network!), mixing scientific models with machine learning will be required to bring AI into these domains in a way that's beyond hype. Physics-informed neural networks [1], differentiable programming [2], universal differential equations [3], SInDy [4], etc. are all ideas directions I think is the next direction for machine learning. [1] https://www.sciencedirect.com/science/article/pii/S0021999118307125 [2] https://arxiv.org/abs/1907.07587 [3] https://arxiv.org/abs/2001.04385 [4] https://www.pnas.org/content/113/15/3932
The Julia Lab (MIT) | DevOps Engineer (+ More) | Onsite or Remote | Full Time | https://julia.mit.edu/ The Julia Lab is a lab in MIT's CSAIL which focuses on the development of software tooling for solving large-scale scientific problems. Named after it's original result, the Julia programming language, the Julia Lab's research ranges from scientific machine learning (SciML) [1] and differentiable programming [2] to large-scale heterogeneous parallel computing [3]. The lab is currently growing in response to new directions and collaborations, and we are looking for both researchers and devops engineers to help with the research activities. We are looking for individuals…
who can help accelerate scientific machine learning workflows with gitops, develop high-precision continuous benchmarking systems, build and wrap essential scientific computing binaries into Julia packages, and help with the deployment of Julia-based software to alternative architectures. Or those with sufficient compiler, numerical computing, and scientific machine learning knowledge may find alternative projects. Experience with Julia is highly preferred. [1] Universal Differential Equations for Scientific Machine Learning, https://arxiv.org/abs/2001.04385 [2] A Differentiable Programming System to Bridge Machine Learning and Scientific Computing, https://arxiv.org/abs/1907.07587 [3] Rapid software prototyping for heterogeneous and distributed platforms, https://www.sciencedirect.com/science/article/abs/pii/S09659...
What are you not sure of with Julia here? There are places in the ecosystem to not be sure of, but this isn't one. Julia has probably the most advanced mathematical optimization right now with JuMP (http://www.juliaopt.org/JuMP.jl/v0.19.2/) and some of the most advanced post-DL machine learning with the full language differentiable programming tools (Zygote, Tracker, ForwardDiff) which have showcased applications like quantum machine learning and neural stochastic differential equations (https://arxiv.org/abs/1907.07587). Some of it is still in flux, but in terms of ecosystem there's a lot of stuff there that you won't find in other languages.
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.