We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network (Weston et al., 2015) but unlike the model in that work, it is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings. It can also be seen as an extension of RNNsearch to the case where multiple computational steps (hops) are performed per output symbol. The flexibility of the model allows us to apply it to tasks as diverse as (synthetic) question answering and to language modeling. For the former our approach is competitive with Memory Networks, but with less supervision. For the latter, on the Penn TreeBank and Text8 datasets our approach demonstrates comparable performance to RNNs and LSTMs. In both cases we show that the key concept of multiple computational hops yields improved results.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Even though this is "Question Answering", it is trained as a classification model. Thus the model will try to come up with one of the top "1000" answers it has seen during the training. This certainly limits the possibility of answers and sometimes returns very weird answer.
It is not for lack of trying that all the top papers in visual question answering end up doing this as a classification task. Results are really poor when it is used as RNN generation, and also extending it more than top 1000 answers does not yield any better results. 87% of the questions in training + validation is within 1000 unique answers.
Latest models have started using more complex form of…
Read the full comment
memory and more tightly integrating the question vectors. One of the top model called DPPNet trains a separate matrix from the question vector (chain of GRUs) to find correspondence on the image filter weights. Their idea is that some question have more relevant areas in the image features. Yet another model DMN+, by Metamind uses dynamic memory network which they build to do language question answers but the extension to images work pretty good.
Surprisingly the models that use visual attention are not the best and I think it is mostly because this kind of model requires even more data and longer training. Just taking 10 different crop of the question image and doing voting of answer beats attention models (based on numbers reported by these papers).
Right now I am working on converting "End to end network" -http://arxiv.org/abs/1503.08895 to this task. I tried working on Neural Turing machine but I could not make it work for this kind of task, but it was mostly because of lack of indepth understanding of NTM.
Any feedback from you guys are welcome.
P.S Thanks fchollet for writing Keras and for this post. Can't wait to try Keras 1.0
Not to be discouraging, but I think research along these lines http://arxiv.org/abs/1503.08895 stands an exponentially better chance of leading towards what OP is talking about. For anyone interested in "building AI", read that paper and all it's references.
The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3].
There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11], handwriting generation [12], playing Go [13], playing Atari [14], speech recognition [15],…
Read the full comment
drawing things [16] and many more.
[1] http://blog.kaggle.com/2015/07/27/taxi-trajectory-winners-in...
[2] http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it...
[3] http://arxiv.org/abs/1403.6652
[4] http://arxiv.org/abs/1502.03240
[5] http://arxiv.org/abs/1411.4734
[6] http://arxiv.org/abs/1409.0473
[7] http://arxiv.org/abs/1502.03044
[8] http://arxiv.org/abs/1410.5401
[9] http://arxiv.org/abs/1506.02216
[10] http://arxiv.org/abs/1507.04808
[11] http://arxiv.org/abs/1503.08895
[12] http://arxiv.org/abs/1308.0850
[13] http://arxiv.org/abs/1412.3409
[14] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
[15] http://www.sciencedirect.com/science/article/pii/S0893608014...
[16] https://www.youtube.com/watch?v=Zt-7MI9eKEo
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.