Skip to content
The methodOpen data
Back to the collection

Show and Tell: A Neural Image Caption Generator

Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan

2014 · arXiv

Computer visionPsychologyMachine learning
Read the paper
HN score
0
Mentions
4
Readers
4
Months
3
What’s in a score?

From the abstract

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

jkldotio
Neutral mention · rule
I think in addition to histograms and features like SIFT, SURF, and others like DAISY[0] that would permit searching with images as a query it would be beneficial just to use a neural network to text index using the classes at very top of a neural network, although you are right some features in layers just below that could be used too as I understand it. You could then use classical text indexing on the text, perhaps with a topic model like LDA. Then an image with a plane in it will be indexed by "plane" via the output of the neural network but would also come first, or in the top results, when using "flight" as the query via a topic model. Ditto for word2vec or para2vec…
Read the full comment
over those words, the benefit being you can bring the knowledge of relations contained in the textual training data, Wikipedia or something else, to bear on the problem. I.e. a golf club and a baseball glove might not be correlated in the neural network that annotated the images but might be correlated in the text based knowledge model trained on Wikipedia and so a query of "sport" might bring both images up. The big players like Google already have caption generation that's capturing relationships between objects.[1] [0]http://scikit-image.org/docs/dev/auto_examples/features_dete... [1]https://arxiv.org/pdf/1411.4555.pdf
Original comment Elasticsearch Based Image Search Using RGB Signatures
danso
Neutral mention · rule
Nice rundown from Mat Honan at Buzzfeed: http://www.buzzfeed.com/mathonan/googles-new-photos-app-is-l... > I have a cousin who I only have a few photos of. In almost all of these, mostly taken last year, she is 30. In another, from 2000, she is 16. But it’s the earliest one in the cluster that’s remarkable. In that picture, my cousin is only 4. Google Photos successfully matched a 4-year-old child’s face to a grown woman’s, with only a lone photo at 16 to tie them together. > Places are also useful. Photos looks at a variety of signals for this, including geodata from photos. But thanks to the wealth of geographic image data Google has amassed, it can also put your old,…
Read the full comment
untagged photos on a map based on landmarks that appear in them. It reliably placed many pictures I had taken in Singapore more than a dozen years ago into a cluster, for example. > This makes Photos very searchable. It’s easy to find places in your pictures — even ones that aren’t geotagged. It’s equally easy to find very specific things, like Labrador retrievers, poodles, and caves. Photos also allows you to search for pictures of people doing things. I found people in poses when I searched for “yoga,” pictures of people with booze, coffee, and Coca-Cola when I searched for “drinking.” Sometimes the results are odd: photos of both wide smiles and crying faces when I searched for “laughing.” Finally, the amazing deep learning imagery algorithms Google and Stanford demonstrated last year [1, 2] are available for consumer benefit. But yes, it really does seem like "Gmail for Photos", in that so much more of your life is now knowable by Google. edit: further reflection...I have probably a TB worth of images of me and my friends over the past five years. I've done a decent job of labeling the folders every time I move files off of my camera and into Lightroom...but the thought of going through all the folders and sorting and tagging them seemed like something I would wait to do in my old age when it's time to make sentimental scrapbooks...Google Photo seems like something that could revolutionize that chore of finding photos of remembrance (for a wedding, funeral, graduation, etc)...on the other hand, it will most definitely have the power to connect everyone to photos taken by their friends, simply through the networks inferred from GMail/G+/etc...I think for the sake of my friends' past assumptions of privacy, I'm going to sit out on using Google Photos for my old photos...but I imagine in the future, we'll be less surprised about how we were auto-tagged in someone else's camera phone. [1] http://arxiv.org/pdf/1411.4555.pdf [2] http://cs.stanford.edu/people/karpathy/deepimagesent/
Original comment Google Photos

Keep following the thread.

More in Computer vision

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score