We describe a new class of learning models called memory networks. Memory networks reason with inference components combined with a long-term memory component; they learn how to use these jointly. The long-term memory can be read and written to, with the goal of using it for prediction. We investigate these models in the context of question answering (QA) where the long-term memory effectively acts as a (dynamic) knowledge base, and the output is a textual response. We evaluate them on a large-scale QA task, and a smaller, but more complex, toy task generated from a simulated world. In the latter, we show the reasoning power of such models by chaining multiple supporting sentences to answer questions that require understanding the intension of verbs.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
It uses a class of RNN they call a MemNN to do something that looks a lot like inference using phrase embeddings.
It's a pretty revolutionary approach - no one seems to have tried anything very similar since the 1990's. The related work section[1] is worth reading.
Interestingly, the Jeff Dean RNN slide deck mentions using RNNs for doing QA, but doesn't expand on what work Google has done in the field. I don't have a URL to hand for that.
They do experiment with loading the MemNN from a set of statements stored as triples, which looks a lot more like a traditional Knowledge Base.
[1] 4 in http://arxiv.org/pdf/1410.3916v8.pdf
Are you sure? I wrote an essay at the end of 2016 about the state of AI research and at the time researchers were demolishing benchmarks like FAIR's bAbI which involved generating answers to questions. I wrote back then about story comprehension and programming robots by giving them stories (we'd now call these prompts).
https://blog.plan99.net/the-science-of-westworld-ec624585e47
bAbI paper: https://arxiv.org/abs/1502.05698
Abstract: One long-term goal of machine learning research is to produce methods that are applicable to reasoning and natural language, in particular building an intelligent dialogue agent. To measure progress towards that goal, we argue for the…
Read the full comment
usefulness of a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human.
So at least FAIR was thinking about making AI that you could ask questions of in natural language. Then they went and beat their own benchmark with the Memory Networks paper:
https://arxiv.org/pdf/1410.3916
Fred went to the kitchen. Fred picked up the milk. Fred travelled to the office.
Where is the milk ? A: office
Where does milk come from ? A: milk come from cow
What is a cow a type of ? A: cow be female of cattle
Where are cattle found ? A: cattle farm become widespread in brazil
What does milk taste like ? A: milk taste like milk
What does milk go well with ? A: milk go with coffee
Where was Fred before the office ? A: kitchen
That was published in 2015. So we can see quite early ChatGPT like capabilities, even though they're quite primitive still.
Semantic hashing by similarity is one way to create more
sensible representations (Re: Proposition 2): http://www.cs.toronto.edu/~rsalakhu/papers/semantic_final.pd...
This would also fit well into memory networks: http://arxiv.org/abs/1410.3916, SDM: http://en.wikipedia.org/wiki/Sparse_distributed_memory
or global workspace model: http://en.wikipedia.org/wiki/Global_Workspace_Theory
This is probably the most important research direction in modern neural network research.
Neural networks are great at pattern recognition. Things like LSTMs allow pattern recognition through time, so they can develop "memories". This is useful in things like understanding text (the meaning of one word often depends on the previous few words).
But how can a neural network know "facts"?
Humans have things like books, or the ability to ask others for things they don't know. How would we build something analogous to that for neural network-powered "AIs"?
There's been a strand of research mostly coming out of Jason Weston's Memory Networks research[1]. This extends on that by…
Read the full comment
using a new form of memory, and shows how it can perform at some pretty difficult tasks. These included graph tasks like London underground traversal.
One good quote showing how well it works:
In this case, the best LSTM network we found in an extensive hyper-parameter search failed to complete the first level of its training curriculum of even the easiest task (traversal), reaching an average of only 37% accuracy after almost two million training examples; DNCs reached an average of 98.8% accuracy on the final lesson of the same curriculum after around one million training examples.
[1] https://arxiv.org/pdf/1410.3916v11.pdf
Jason Weston (Facebook) etc's work on Memory Nets[1]
Anything to do with transfer learning (the 2013 Zero-Shot Learning Through Cross-Modal Transfer[2] paper is a good place to start)
The increasing amount of demos around using NNs to generate "things" that look kinda-almost "intelligent". I can't point at a paper, but Andrej Karpathy demo of generating Shakespere-like writing, "Wikipedia" pages and "C" code in The Unreasonable Effectiveness of Recurrent Neural Networks[3] is the kind of thing I'm talking about.
The beginnings of work around goal-seeking. The (now Google) DeepMind Atari demo[4] and Marl/O[5]
Finally, the work being done on making this stuff usable by programmers (Torch etc).
[1] http://arxiv.org/abs/1410.3916
[2] http://arxiv.org/pdf/1301.3666.pdf
[3] http://karpathy.github.io/2015/05/21/rnn-effectiveness/
[4] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
[5] https://www.youtube.com/watch?v=qv6UVOQ0F44
Abstract If future net-zero emissions energy systems rely heavily on solar and wind resources, spatial and temporal mismatches between resource availability and electricity demand may challenge system reliability.