Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
As a complement to all the other comments so far: It is plausible that any goal directed inference must "compress" the raw data into a small number of bits required to make a decision on the goal. For physicists "renormalization group flow" (RG) is nothing but a theory behind that process of "simplifying" a theory enough to make predictions (towards a specific goal). It is also natural that this compression process is iterative. There are many implementations of RG that fall within the framework, each with their own pros and cons, which makes each of them suitable for solving different problems.
One of the more recent examples is the "Multi-scale Entanglement…
Read the full comment
Renormalization Ansatz" (MERA) tensor network which accounts quantum entanglement in the RG framework (none of the previous formulations did, to my knowledge). In my opinion, that is a significant qualitative breakthrough, whose consequences are starting to bear fruit, and will continue for the next decade. OP compares MERA with RBMs and claims that they're similar.
For an information theoretic perspective of this relation between deep learning and RG, refer to this fascinating paper by Naftali Tishby and Noga Zaslavsky, that brings together ideas from different fields: https://arxiv.org/abs/1503.02406
Regarding the implications:
1. For deep learning: We might have a shot at coming up with a theory for how DNNs work. This will help us design (much) better DNNs because today's designs are pretty ad-hoc and might be far from optimal. It might also give us some explanator power in reasoning about the way DNNs work.
2. For physics: It's not clear what the payoffs would be. There has been some recent work trying to use DNNs to solve physics-ish problems (eg: guessing the phase of the Ising mode, given MCMC generated samples). The hope is that in trying to study this relation, we uncover a better understanding of how RG works. That would have potentially tremendous consequences, for every thing from designing superconductors to understanding quantum gravity.
PS: I'm a physicist and I've been exploring this area over the last year or so; if anyone is interested in discussing details, feel free to ping me.
Gotcha. I'm happy to do the trace as it likely would be fruitful for me.
Do you have a link to a specific post you're thinking of? It's likely going to be a Tishby-like (the classic paper from 2015
{with much more work going back into the early aughts, just outside of the NN regime IIRC}: https://arxiv.org/abs/1503.02406) lineage, but I'm happy to look to see if it's novel.
if you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening...
but some of his key claims are probably contradicted by empirical results.
the original paper:
https://arxiv.org/abs/1503.02406
a refutation:(with very harsh/unprofessional words by tishby in counter-refutation)
https://openreview.net/forum?id=ry_WPG-A-¬eId=ry_WPG-A-
Here's an interesting follow up from a noted researcher (Tali Tishby) with one foot in the stat mech camp, and one in the information theory camp: https://arxiv.org/pdf/1503.02406.pdf
I am working on a project (using Julia) to "learn" an efficient representation of scale invariant many-body quantum states from a microscopic Hamiltonian. The particular structure being learned is supposed to be highly efficient both in representational complexity and computational complexity for predicting physical observables. (Work in progress at https://github.com/sivark/jumera/ )
Also reading/thinking about ideas at the heart of theoretical physics and machine learning. In particular, I've spent the last several months in trying to understand the following two papers in increasing levels of depth
1. https://arxiv.org/abs/1410.3831
2. https://arxiv.org/abs/1503.02406
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.