Yoon Kim
2014 · arXiv
We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
Thanks for this, I'll have to take a look at FastText. I've been using word2vec before turning it into a matrix and running it through a CNN. I based it off of Yoon Kim's[0] work. I haven't had much luck though on my 92-class problem. Maybe Fast Text will work better, although I think there are a lot of improvements my model can have still. [0] https://arxiv.org/abs/1408.5882
A few remarks: In the "Doing bad digital humanities with color vectors", if you consider colors as 3D vectors, which they do, you'll see that summing enough uniformly sampled vectors always gives medium browns, because that's the color in the middle of the colorspace. Instead, you should model colors in a polar space and sum vectors in that space. This will prevent going inside the sphere and losing color saturation. It's explained quite well here in the Interpolation section: http://www.inference.vc/high-dimensional-gaussian-distributi... If you want to understand contemporary use of words embeddings in ML, a nice simple model is explained, with full code, here: https://blog.keras.io/using-pre-trained-word-embeddings-in-a... The original model comes from Kim 2014, https://arxiv.org/abs/1408.5882 It's a very neat use of CNNs for language processing, instead of the more popular RNNs/LSTMs. CNNs have the advantage of training much faster.
Yes I have. Convolutional Nets work well for simpler cases. See [Yoon Kim 2014](https://arxiv.org/abs/1408.5882). FYI "Tagging" in NLP is more used for labelling individual words (verbs, nouns, pronouns, etc). "Classification" is a better keyword here.
Surprising there is no mention of any CNN based methods for NLP. A different approach that seems to provide results on par with the state of the art in some NLP tasks [1]. [1] http://arxiv.org/abs/1408.5882
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.