A capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. Active capsules at one level make predictions, via transformation matrices, for the instantiation parameters of higher-level capsules. When multiple predictions agree, a higher level capsule becomes active. We show that a discrimininatively trained, multi-layer capsule system achieves state-of-the-art performance on MNIST and is considerably better than a convolutional net at recognizing highly overlapping digits. To achieve these results we use an iterative routing-by-agreement mechanism: A lower-level capsule prefers to send its output to higher level capsules whose activity vectors have a big scalar product with the prediction coming from the lower-level capsule.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Direct link to the Google Brain paper, which is well worth a read:
https://dl.acm.org/citation.cfm?id=3321441 (click on "PDF" link to read)
Abstract: "In this paper we argue that systems for numerical computing are stuck in a local basin of performance and programmability. Systems researchers are doing an excellent job improving the performance of 5-year-old benchmarks, but gradually making it harder to explore innovative machine learning research ideas. We explain how the evolution of hardware accelerators favors compiler back ends that hyper-optimize large monolithic kernels, show how this reliance on high-performance but inflexible kernels reinforces the dominant style…
Read the full comment
of programming model, and argue these programming abstractions lack expressiveness, maintainability, and modularity; all of which hinders research progress. We conclude by noting promising directions in the field, and advocate steps to advance progress towards high-performance general purpose numerical computing systems on modern accelerators."
The main example the authors use to illustrate these issues is capsule networks, first proposed two years ago.[a]
To date, no one has been able to develop a high-performance implementation of capsule networks. At present, the best-performing implementations in Tensorflow and PyTorch must copy, rearrange, and materialize to memory two orders of magnitude more data than necessary, due to the issues raised by the authors. See sections 1 and 2 of the paper for the gory details.
Two orders of magnitude. That is pathetic.
[a] https://arxiv.org/abs/1710.09829
I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list
https://twitter.com/vrungta/status/1623343807227105280
Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023)
1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain)
2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal)
3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain)
4. "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks" (2016) -…
Read the full comment
https://arxiv.org/abs/1511.06434 (University of Montreal)
5. "ImageNet Classification with Deep Convolutional Neural Networks" (2012) - https://papers.nips.cc/paper/4824-imagenet-classification-wi... (University of Toronto)
6. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2018) - https://arxiv.org/abs/1810.04805 (Google)
7. "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (2019) - https://arxiv.org/abs/1907.11692 (Facebook AI)
8. "ELMo: Deep contextualized word representations" (2018) - https://arxiv.org/abs/1802.05365 (Allen Institute for Artificial Intelligence)
9. "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" (2019) - https://arxiv.org/abs/1901.02860 (Google AI Language)
10. "XLNet: Generalized Autoregressive Pretraining for Language Understanding" (2019) - https://arxiv.org/abs/1906.08237 (Google AI Language)
11. T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (2020) - https://arxiv.org/abs/1910.10683 (Google Research)
12. "Language Models are Few-Shot Learners" (2021) - https://arxiv.org/abs/2005.14165 (OpenAI)
You made me think of the Margaret Thatcher Illusion, for which the best example I found was Dr. Phil Plait[0]. Funny thing, the idea of Capsule Networks (Hinton et al)[1, 2] seems to tackle some of these issues, though it's they're a bit young yet and more more work.
[0] https://www.opticalspy.com/opticals/dr-phil-plait-optical-il...
[1] https://arxiv.org/abs/1710.09829
[2] https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...
I wonder how well capsule networks could deal with this, considering they're more robust than traditional CNNs towards other sorts of adversarial attacks. My guess is that one-pixel changes are going to do very little to alter pose information (though they will still alter the confidence of the feature existing), and so caps nets should be more robust here as well. Does anyone know if my intuition is correct?
Link to the capsule network paper, for those who haven't heard of it: https://arxiv.org/abs/1710.09829
Is the mechanism of 'Arc' gene related to the dynamic routing process of Capsule networks [1, 2]?
[1] https://kndrck.co/posts/capsule_networks_explained/
[2] https://arxiv.org/abs/1710.09829
Dynamic Routing Between Capsules
the long awaited paper from Geoffrey Hinton (although Sara Sabour is principal author, Hinton has been hinting at this for a while now) on an alternative to back-propagation
https://arxiv.org/abs/1710.09829
I sympathize with this being in a similar professional situation and having once been introduced to this field via Hofstadter. His musings are usually on a distinctly symbolic level however, and though it is not my intention to defend a kind of results-only pragmatic approach, I do concern myself with strategies for progressively reaching better models of intelligence and their realization in contemporary hardware in particular. What, for instance, is your opinion about Hinton's recently published capsules idea?
https://arxiv.org/abs/1710.09829
https://openreview.net/forum?id=HJWLfGWRb
Capsule networks[1] have literally not been used on anything larger than what is considered a toy dataset for computer vision yet. Whilst there is potential there I'd argue it's too early to ascribe extreme levels of excitement towards it.
[1]: https://arxiv.org/abs/1710.09829
> 2% is HUGE at this point
Hell, on MNIST 0.14% is huge. Geff Hinton created an entire new architecture to get 0.25% error, which is far better than the baseline 0.39% error [1].
To be fair, he did reduce the error by 35%.
[1]:https://arxiv.org/pdf/1710.09829.pdf
The only thing is that, when this presentation was given, it seems they hadn't worked much more than MNIST (so the new thing now would be the toys-recognition net).
The paper however details their experiments on Cifar-10 and other datasets in the discussion section.
They don't produce good enough results but the paper proposes certain hypotheses for the poorer performance and that it could be overcome in the future.
Arxiv-link: https://arxiv.org/abs/1710.09829
This is trueish, but there is more to it than that.
It is true for sure that absolute performance on MNIST isn't the most interesting thing in the world.
But when introducing a new tool or technique being able to show competitive performance on MNIST is a good way to show that it isn't an entirely useless thing.
I'd note that recent Sabour, Frosst and Hinton paper[1] (where they finally got Hinton's capsules to work) spends most of the paper analyzing how it performs on MNIST, and only a short section on other datasets.
I assume I don't need to point out that Geoff Hinton does know a little about deep learning, and if he thinks submitting a NIPS paper on MNIST is acceptable in 2017 then I'm not going to argue too hard against it.
[1] https://arxiv.org/pdf/1710.09829.pdf
I'm surprised there is no mention of capsules and capsule-routing algorithms.
Capsules are groups of neurons that represent discrete entities in different contexts. For example, a 4x4 pose matrix is a capsule representing a particular object in different orientations seen from different viewpoints. Similarly, a subword embedding can be seen as a capsule with vector shape representing a particular subword in different natural language contexts. More generally, a capsule can have any shape, but it always represents only one entity in some context.
In certain new capsule-routing algorithms -- e.g., EM routing[a], Heinsen routing[b], dynamic routing[c], to name a few off the…
Read the full comment
top of my head[d] -- each capsule can activate or not depending on whether the entity it represents is detected or not in the context of input data.
Models using these algorithms therefore make it possible for human beings to interpret model behavior in terms of capsule activations -- e.g., "the final layer predicts label 2 because capsules 7, 23, and 41 activated the most in the last hidden routing layer."
While these new routing algorithms are not yet widely used, in my humble opinion they present a promising avenue of research for building models that are explainable and/or enable assignment of causality at high levels of function composition.
--
[a] https://research.google/pubs/pub46653/
[b] https://arxiv.org/abs/1911.00792
[c] https://arxiv.org/abs/1710.09829
[d] If you're aware of other routing algorithms that can similarly activate/deactivate capsules, please post a link to the paper or code here.
Keno: first of all, let me give a big public thank you to you and your colleagues. (For those here who don't know, Keno is listed as one of the authors of the paper, and works closely with Mike Innes, lead author and also lead developer of Zygote. Mike is also an active member of HN.)
Second, let me bring up what I think is a significant issue. My perception is that most deep learning researchers and practitioners -- that includes me -- tend to iterate very rapidly over new ideas. We want to test ideas in code as quickly as possible, and in practice we will not invest the time and effort necessary to figure out how to write cache-optimized kernels (e.g., for GPUs) every…
Read the full comment
time we might need one. In fact, I'd say the default attitude is that it doesn't even make sense for us to use (what we perceive as slow) automated kernel-writing tools to search for and compile optimized kernels. In practice, it always seems faster and easier (from a developer-time standpoint) to write code that leverages existing, inflexible, prebuilt, handtuned kernels that have already proven to work well, are fast, and are already nicely integrated with frameworks like PyTorch and TensorFlow.
There was a good discussion of this topic in the forums a few weeks ago[a] in response to a recent paper published by some folks at Google in which they make a compelling case that this issue is holding back AI research.[b] As an example, they use capsule networks[c], the implementations of which have proven difficult to optimize (e.g., they copy a lot more data around than is strictly necessary, in order to slice and reshape data in a manner that is compatible with preexisting hardware-accelerator kernels).
What are your thoughts on this? Will Zygote provide any improvements or advantages on this front?
--
[a] https://news.ycombinator.com/item?id=20301619
[b] https://dl.acm.org/citation.cfm?id=3321441
[c] https://arxiv.org/abs/1710.09829
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.