Skip to content
The methodOpen data
Back to the collection

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Kelvin Xu, Jimmy Ba, Ryan Kiros

Show all 8 authors

Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio

2015 · arXiv

Statistics & probabilityMachine learningHuman–computer interaction
Read the paper
HN score
0
Mentions
6
Readers
5
Months
6
What’s in a score?

From the abstract

Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

ahartman00
Neutral mention · rule
Not sure why you were downvoted, I'm assuming this was a good faith question. I'm going to try to answer your question, but I'm still learning, so take this with a grain of salt. Some models have attention, but it is not a requirement. In models that do have attention, it may or may not be simple. In image captioning for example, you can do just that. See [1] for some pictures of this happening. But in this example, they are stepping through a caption, and seeing where the attention is focused for each word. Works fine for a short caption. Videos are just many images and sound, but already this is going to be more difficult. For natural language processing, you will be…
Read the full comment
stepping through very many words. Some models use characters instead of words. Not sure how you could even make sense of that. I have seen people look at the attention in machine translation. Not sure how it would work for sentiment analysis. So you aren't exactly wrong, but you are greatly over simplifying this for the models where it is possible(based on my understanding). Overall, machine learning is very much a black box, despite efforts to the contrary. It also doesn't solve some of the other problems. For example, knowing the model is focusing on zip code doesn't help you remove bias that comes from the data. 1. https://arxiv.org/pdf/1502.03044.pdf
Original comment Asking the Right Questions About AI
hiddencost
Neutral mention · rule
https://arxiv.org/abs/1311.2901 Check out figure 2. The network learns composition from fundamental shapes and gradients to compositional ones. It's kinda awe inspiring. Not NN specific, but some more work on explanations: https://homes.cs.washington.edu/~marcotcr/blog/lime/ Visualization of attention mechanisms is pretty cool for explanations: https://arxiv.org/pdf/1502.03044.pdf http://torch.ch/blog/2015/09/21/rmva.html
Original comment Understanding deep learning requires rethinking generalization
kastnerkyle
Neutral mention · rule
There are a number of papers tackling a similar task [0][1][2] for anyone who is interested. There isn't enough information to tell exactly what is going on with Deepgram, but one way to approach this would be to construct a shared embedding space for words/phrases and speech. These types of embedding spaces are powerful [3][4][5], but not magic. Cool demo, looking forward to seeing more detail about what is going on. However I would quibble with the STT WER quoted above. Maybe in noisy environments with unknown speakers (and no voice normalization) this is accurate, but the kinds of clean speech in the demo perform really well in modern recognition engines (on benchmark…
Read the full comment
data, to be fair c.f. MSR 6.3% and IBM at ~6.9%). Most word searches over speech to text work over soft matches (or ideally beam search over most likely partial phoneme/word part matches), rather than hard matches so it seems like a bit of a straw man comparison in this case. [0] http://research.google.com/pubs/pub42543.html [1] https://arxiv.org/abs/1510.01032 [2] https://sigport.org/sites/default/files/gloveNNLM_kaudhkhasi... [3] https://arxiv.org/abs/1502.03044 [4] http://www-personal.umich.edu/~reedscot/files/icml2016.pdf [5] https://arxiv.org/abs/1411.2539
Original comment Deepgram – Find Damning Soundbites
waterlesscloud
Neutral mention · rule
Looks like it's the home page of Ryan Kiros at the University of Toronto. Here's a paper with his name on it with some related work, though I suspect this particular output is from a later project. http://arxiv.org/abs/1502.03044
Original comment Neural Artistic Captions
kastnerkyle
Neutral mention · rule
The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3]. There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11], handwriting generation [12], playing Go [13], playing Atari [14], speech recognition [15],…
Read the full comment
drawing things [16] and many more. [1] http://blog.kaggle.com/2015/07/27/taxi-trajectory-winners-in... [2] http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... [3] http://arxiv.org/abs/1403.6652 [4] http://arxiv.org/abs/1502.03240 [5] http://arxiv.org/abs/1411.4734 [6] http://arxiv.org/abs/1409.0473 [7] http://arxiv.org/abs/1502.03044 [8] http://arxiv.org/abs/1410.5401 [9] http://arxiv.org/abs/1506.02216 [10] http://arxiv.org/abs/1507.04808 [11] http://arxiv.org/abs/1503.08895 [12] http://arxiv.org/abs/1308.0850 [13] http://arxiv.org/abs/1412.3409 [14] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf [15] http://www.sciencedirect.com/science/article/pii/S0893608014... [16] https://www.youtube.com/watch?v=Zt-7MI9eKEo
Original comment A Beginner’s Guide to Restricted Boltzmann Machines

Keep following the thread.

More in Statistics & probability

More from the same topics.