Skip to content
The methodOpen data
Back to the collection

Pointer Networks

Oriol Vinyals, Meire Fortunato, Navdeep Jaitly

2015 · arXiv

AlgorithmsStatistics & probability
Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

We introduce a new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence. Such problems cannot be trivially addressed by existent approaches such as sequence-to-sequence and Neural Turing Machines, because the number of target classes in each step of the output depends on the length of the input, which is variable. Problems such as sorting variable sized sequences, and various combinatorial optimization problems belong to this class. Our model solves the problem of variable size output dictionaries using a recently proposed mechanism of neural attention. It differs from the previous attention attempts in that, instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, it uses attention as a pointer to select a member of the input sequence as the output. We call this architecture a Pointer Net (Ptr-Net). We show Ptr-Nets can be used to learn approximate solutions to three challenging geometric problems -- finding planar convex hulls, computing Delaunay triangulations, and the planar Travelling Salesman Problem -- using training examples alone. Ptr-Nets not only improve over sequence-to-sequence with input attention, but also allow us to generalize to variable size output dictionaries. We show that the learnt models generalize beyond the maximum lengths they were trained on. We hope our results on these tasks will encourage a broader exploration of neural learning for discrete problems.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

cute_boi
Neutral mention · rule
No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/ ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385 Multi-Scale Context Aggregation by Dilated Convolutions - https://arxiv.org/abs/1511.07122 Identity Mappings in Deep Residual Networks -…
Read the full comment
https://arxiv.org/abs/1603.05027 Recurrent Neural Network Regularization - https://arxiv.org/abs/1409.2329 Deep Speech 2: End-to-End Speech Recognition in English and Mandarin - https://arxiv.org/abs/1512.02595 Order Matters: Sequence to Sequence for Sets - https://arxiv.org/abs/1511.06391 Neural Machine Translation by Jointly Learning to Align and Translate - https://arxiv.org/abs/1409.0473 Pointer Networks - https://arxiv.org/abs/1506.03134 Attention Is All You Need - https://arxiv.org/abs/1706.03762 The Annotated Transformer - https://nlp.seas.harvard.edu/annotated-transformer/ Neural Turing Machines - https://arxiv.org/abs/1410.5401 A Simple Neural Network Module for Relational Reasoning - https://arxiv.org/abs/1706.01427 Relational Recurrent Neural Networks - https://arxiv.org/abs/1806.01822 Neural Message Passing for Quantum Chemistry - https://arxiv.org/abs/1704.01212 Scaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361 GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism - https://arxiv.org/abs/1811.06965 Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077 The First Law of Complexodynamics - https://scottaaronson.blog/?p=762 Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903 Kolmogorov Complexity - https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X Variational Lossy Autoencoder - https://arxiv.org/abs/1611.02731 Machine Super Intelligence - https://www.vetta.org/documents/Machine_Super_Intelligence.p...
Original comment 30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format
BenderV
Neutral mention · rule
Well, there is a variety of possible answers here. Giving the direct integer ("1", "23") to a simple deep network won't work. - RNN with LSTM might be a better approach, since it will more based on the value of the ordered bits than their "disposition", and could scale to an arbitrary high number. - @zardo mentioned Pointer Network (https://arxiv.org/abs/1506.03134). It looks to solve this kind of problems but honestly I'm discovering it now. - Give the base 3 and 5 of the input would be the fastest solution. (but since it's kind of hardcoding part of the solution, it's more a trick than a general solution). Can anyone correct me what if I got something wrong here? Did I missed something?
Original comment Fizz Buzz in Tensorflow

Keep following the thread.

More in Algorithms

More from the same topics.