One Model To Learn Them All

merely discover statistical regularities that relate words to other words at multiple, hierarchical levels of composition. In Hofstadter's words, "there's no attempt to create internal structures that could be thought of as ideas, images, memories, or experiences. Such mental etherea are still far too elusive to deal with computationally." He is right. That said, AI researchers are aware of these limitations, and are exploring possible ways to overcome them. An early, crude example of such research is the multi-modal model of "One Model to Learn Them All" (https://arxiv.org/abs/1706.05137), which is trained to learn to perform multiple image-recognition, language-translation, image-captioning, speech-recognition, and language-parsing tasks at the same time, using representations that are shared by all tasks. While these early research efforts fall far short of the kind of "understanding of the world" Hofstadter shows is necessary for human-level language translation, it's encouraging to see AI researchers actively looking for ways to move beyond the mere discovery of 'hierarchical statistical regularities' that relate words to other words. This is an exciting time for AI research. [a] https://en.wikipedia.org/wiki/Douglas_Hofstadter

attention mechanism, and sparsely-gated layers. Each of these computational blocks is crucial for a subset of the tasks we train on. Interestingly, even if a block is not crucial for a task, we observe that adding it never hurts performance and in most cases improves it on all tasks. We also show that tasks with less data benefit largely from joint training with other tasks, while performance on large tasks degrades only slightly if at all. https://arxiv.org/abs/1706.05137 EDIT: And code: https://github.com/tensorflow/tensor2tensor