Skip to content
The methodOpen data
Back to the collection

Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

Michael M. Bronstein, Joan Bruna, Taco Cohen, Petar Veličković

2021 · arXiv

Machine learningMathematicsLanguage models
Read the paper
HN score
1
Mentions
7
Readers
5
Months
6
What’s in a score?

From the abstract

The last decade has witnessed an experimental revolution in data science and machine learning, epitomised by deep learning methods. Indeed, many high-dimensional learning tasks previously thought to be beyond reach -- such as computer vision, playing Go, or protein folding -- are in fact feasible with appropriate computational scale. Remarkably, the essence of deep learning is built from two simple algorithmic principles: first, the notion of representation or feature learning, whereby adapted, often hierarchical, features capture the appropriate notion of regularity for each task, and second, learning by local gradient-descent type methods, typically implemented as backpropagation. While learning generic functions in high dimensions is a cursed estimation problem, most tasks of interest are not generic, and come with essential pre-defined regularities arising from the underlying low-dimensionality and structure of the physical world. This text is concerned with exposing these regularities through unified geometric principles that can be applied throughout a wide spectrum of applications. Such a 'geometric unification' endeavour, in the spirit of Felix Klein's Erlangen Program, serves a dual purpose: on one hand, it provides a common mathematical framework to study the most successful neural network architectures, such as CNNs, RNNs, GNNs, and Transformers. On the other hand, it gives a constructive procedure to incorporate prior physical knowledge into neural architectures and provide principled way to build future architectures yet to be invented.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

PartiallyTyped
Recommendation · rule
I recommend against DL by Goodfellow. At this point it is pretty much outdated. Actually, anything specific to NNs is already outdated by release. You'd need the following background: - Linear Algebra - Multivariate Calculus - Probability theory && Statistics Then you need a decent ML book to get the foundations of ML, you can't go wrong with either of these: - Bishop's Pattern Recognition - Murphy's Probabilistic ML. - Elements of statistical learning. You can supplement Murphy's with the advanced book. Elements is a pretty tough book, consider going through "Introductions to Elements of statistical learning". Bishop and Murphy include foundational topics in…
Read the full comment
mathematics. I'd also recommend doing some abstract algebra, but it's not a prerequisite. For NN methods, some recommendations: - https://paperswithcode.com/methods/category/regularization - https://paperswithcode.com/methods/category/stochastic-optim... - https://paperswithcode.com/methods/category/attention-mechan... - https://paperswithcode.com/paper/auto-encoding-variational-b... For something a little bit different but worth reading given that you have the prerequisite mathematical maturity - Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges | https://arxiv.org/abs/2104.13478
Original comment Ask HN: What are the foundational texts for learning about AI/ML/NN?
r-bryan
Neutral mention · rule
Check out this 156-page tome: https://arxiv.org/abs/2104.13478: "Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges" The intro says that it "...serves a dual purpose: on one hand, it provides a common mathematical framework to study the most successful neural network architectures, such as CNNs, RNNs, GNNs, and Transformers. On the other hand, it gives a constructive procedure to incorporate prior physical knowledge into neural architectures and provide principled way to build future architectures yet to be invented." Working all the way through that, besides relearning a lot of my undergrad EE math (some time in the previous century), I learned a whole new bunch of differential geometry that will help next time I open a General Relativity book for fun.
Original comment The maths you need to start understanding LLMs
smath
Neutral mention · rule
also relevant to 3d modeling of molecules using in drug discovering modeling -- and a subset of these authors have published along those lines more recently -- for e.g. https://arxiv.org/abs/2104.13478
Original comment Spherical CNNs (2018)
PartiallyTyped
Neutral mention · rule
You may be interested in [1] where they go to a great extend to show that the convolution operation that we consider in DL is the dual of fourier series [2]. [1] https://geometricdeeplearning.com [2] https://arxiv.org/pdf/2104.13478.pdf page 27 (23 if you count book pages).
Original comment Why train when you can optimize?

Keep following the thread.

More in Machine learning

More from the same topics.

1994arXiv

On proof and progress in mathematics

William P. Thurston

In response to Jaffe and Quinn [math.HO/9307227], the author discusses forms of progress in mathematics that are not captured by formal proofs of theorems, especially in his own work in the theory of foliations and geometrization of 3-manifolds and dynamical systems.

Mathematics
29 mentions21 months
9HN score