Jordan Guergiuev, Timothy P. Lillicrap, Blake A. Richards
2016 · arXiv
Deep learning has led to significant advances in artificial intelligence, in part, by adopting strategies motivated by neurophysiology. However, it is unclear whether deep learning could occur in the real brain. Here, we show that a deep learning algorithm that utilizes multi-compartment neurons might help us to understand how the brain optimizes cost functions. Like neocortical pyramidal neurons, neurons in our model receive sensory information and higher-order feedback in electrotonically segregated compartments. Thanks to this segregation, the neurons in different layers of the network can coordinate synaptic weight updates. As a result, the network can learn to categorize images better than a single layer network. Furthermore, we show that our algorithm takes advantage of multilayer architectures to identify useful representations---the hallmark of deep learning. This work demonstrates that deep learning can be achieved using segregated dendritic compartments, which may help to explain the dendritic morphology of neocortical pyramidal neurons.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
Look into capsules, also [1-3] can give you an idea of what can be explored. Evolving NNs [4] would probably benefit the most from NN primitives research. IMO, the best source of ideas for novel primitives is neocortex [5, 6]. [1] https://arxiv.org/abs/1904.03955 [2] https://arxiv.org/abs/1811.11168 [3] https://arxiv.org/abs/1905.02961 [4] https://www.nature.com/articles/s42256-018-0006-z [5] https://arxiv.org/abs/1610.00161 [6] https://arxiv.org/abs/1511.00083
There was a rather interesting talk at the Forbes Under 30 summit last month on this, "How The Brain is Inspiring AI" [1], but I cannot find the video. Lavin argued neuro and cognitive science -inspiration, not derivation, is useful at varying levels of abstraction, i.e. Marr's levels [2]. He showcased three works at the respective levels: Tenenbaum et al at MIT [3], some Vicarious research [4], and Blake Richards' work towards "DL with segregated dendrites" [5]. [1] https://www.forbes.com/forbes-live/event/a-i-machine-learnin... [2] http://blog.shakirm.com/2013/04/marrs-levels-of-analysis/ [3] https://cbmm.mit.edu/about/people/tenenbaum [4] https://www.vicarious.com/2017/10/26/common-sense-cortex-and... [5] https://arxiv.org/abs/1610.00161
An AI "winter" is a long period in which [edit: funding is cut because...] researchers are in disbelief about having a path to real intelligence. I think that is not the case at this time, because we have (or approaching) adequate tools to rationally dismiss that disbelief. The current AI "spring" has brought back the belief that connectionism may by the ultimate tool to explain the human brain. I mean you can't deny that DL models of vision look eerily like the early stages in visual processing in the brain (which is a very large part of it). Even if DL researchers lose their path in search for "true AI", the neuroscientists can keep probing the blueprint to find new…
clues to its intelligence. Even AI companies are starting to create plausible models that link to biology. So at this time, it's unlikely that progress will be abandoned any time soon. E.g. https://arxiv.org/abs/1610.00161 https://arxiv.org/abs/1706.04698 https://www.ncbi.nlm.nih.gov/pubmed/28095195
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.
Scott Aaronson
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.