An exact mapping between the Variational Renormalization Group and Deep Learning

Thanks for this additional link, which really underscores for me at least how you're right about patterns in circuits being a better abstraction layer for capturing interesting patterns than topological manifolds. I wasn't familiar with the term "equivariance" but I "woke up" to this sort of approach to understanding deep neural networks when I read this paper, which shows how restricted boltzman machines have an exact mapping to the renormalization group approach used to study phase transitions in condensed matter and high energy physics: https://arxiv.org/abs/1410.3831 At high enough energy, everything is symmetric. As energy begins to drain from the system, eventually every symmetry is broken. All fine structure emerges from the breaking of some symmetries. I'd love to get more in the weeds on this work. I'm in my own local equilibrium of sorts doing much more mundane stuff.

it's quite plausible that at the fringe/edge there is research being done that involves differential geometry - hell there's this paper https://arxiv.org/pdf/1410.3831v1.pdf that's occupies very rarefied air, but that's not what the person to whom i'm responding to claims (that somewhere someone there's a relationship): >DG is the foundation of all modern AI/ML/DL which is just blather spoken in a tone of confidence in order to sound very smart.

Thank you for posting this. Similar thoughts have been percolating in my stream-of-consciousness for a while, especially after coming across an earlier version of Tegmark & Lin's paper some months ago. My take is that deep neural nets work so well in practice because not all distributions of natural data are equally likely (and therefore, the no-free-lunch theorem(s), which assumes all distributions are equally likely, doesn't hold in the real world!); and that, in turn, is because the distribution of natural data is a consequence of the laws of Physics / symmetries of the universe in which we happen to live. PS. You will enjoy the following papers too: "An exact mapping between the Variational Renormalization Group and Deep Learning" - https://arxiv.org/abs/1410.3831 ; and "Mutual Information, Neural Networks and the Renormalization Group" - https://arxiv.org/abs/1704.06279 .

Part of me says that you are correct on all of these points. However I think you went too far on 3. Experimentation does not always follow theory in science. I would argue that many of the great discoveries of physics in the 20th century followed directly from startling results of experiments conducted at the end of the 19th century (the photoelectric effect for example). I agree that there seems to be an awful lot of experimenting going on and that hardware and large datasets have helped tremendously but there are also many researchers poking and prodding at deep learning theory [1, 2]. So from a glass-half-full perspective we have rapid (sometimes iterative) experimentation coupled with yawning gaps in theory to explain surprising results. In other words, the opposite of previous AI booms and a big reason to be optimistic despite all the hype. [1] https://arxiv.org/pdf/1410.3831v1.pdf [2] https://arxiv.org/abs/1608.08225

I am working on a project (using Julia) to "learn" an efficient representation of scale invariant many-body quantum states from a microscopic Hamiltonian. The particular structure being learned is supposed to be highly efficient both in representational complexity and computational complexity for predicting physical observables. (Work in progress at https://github.com/sivark/jumera/ ) Also reading/thinking about ideas at the heart of theoretical physics and machine learning. In particular, I've spent the last several months in trying to understand the following two papers in increasing levels of depth 1. https://arxiv.org/abs/1410.3831 2. https://arxiv.org/abs/1503.02406

You should watch Max Tegmark's talk "Connections between Physics and Deep Learning" if this interests you [1]. Additionally, a paper that has everyone excited about deep connections between the mathematical analysis of physical systems and the hierarchical feature learning paradigm speaks of the connection in terms of the Renormalization Group [2]. Regardless of it's practical utility, the philosophical implications do tickle the intellect. On a dreamy note, I wonder if it would be possible to draw Category-theoretic parallels between some physical theories and statistical learning theory. There is so much to learn, and I am trying my best to teach myself (on the side) the mathematics that they don't teach in my CS grad school. So much to learn, such little time. :) [1] https://youtu.be/5MdSE-N0bxs [2] http://arxiv.org/abs/1410.3831

Love this curious and open-minded exploration of how this stuff works. The pyramid strategy loosely tracks with renormalization group theory, which has been formally studied for years as a method of interpreting machine learning models: https://arxiv.org/abs/1410.3831 I love the convergence we're seeing in the use of models from different fields to understand machine learning, fundamental physics, and human consciousness. What a time to be alive.