Machine Learning Systems are Stuck in a Rut

As much as many of us would like it to be, the kind of data science work you see Scala used for is a pretty small part of what Julia is used for. I think a big part of that is because DS rarely involves writing fast numeric kernels or hot inner loops, i.e. user code that needs to do numeric stuff quickly. This is in large part because very large organizations have poured untold millions into libraries that already handle this (e.g Spark). In domains where this has not happened or that have more bespoke requirements (e.g. modelling and simulation), something like Julia is far more compelling. That's not to say it's not viable, but unless more practitioners start feeling stuck in a rut [1] I don't see the mindshare changing dramatically. [1] https://dl.acm.org/doi/10.1145/3317550.3321441

The Dex authors are also core Autograd & JAX authors, so they're not unaware of the benefits of the dense Numpy-style APIs. The challenge is that many numerical kernels and algorithms are just not easily or naturally expressed in this vectorized language. (Think sparsity, irregular access patterns, control flow, etc.) Current languages, runtimes and hardware currently force most work into this dense, rectangular, vectorized style to achieve reasonable performance, but we would like our ideas to not always remain stuck in this "rut" [1]. Dex is one exploration of how we can improve expressiveness and generality while still allowing for performant compiled artifacts. [1]: https://dl.acm.org/doi/10.1145/3317550.3321441

You would think so, but the problem is more complex than it appears to be: https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=... There's a big "impedance mismatch" between (1) "programmability" (by which I mean, being able to write high-level code that under-the-hood requires dynamic modification, reshaping, and/or combination-optimization of those adjacency matrices you mention, without worrying about performance) and (2) existing infrastructure (frameworks that rely on highly optimized computation "kernels" that cleverly exploit the memory layout and other features of accelerated hardware, i.e., GPUs/TPUs).

Unfortunately, the problem is more complex than that: https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=... (I oversimplified things in my comment above).

The problem is more complex than "just" hardware. If I may paraphrase a recent paper out of Google Brain: the current systems (hardware and software) we use for for numerical computing are "stuck in a local basin of performance and programmability."[a] [a] https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=...