Jeremy Howard, Sebastian Ruder
2018 · arXiv
Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100x more data. We open-source our pretrained models and code.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
5 selected excerpts · 5 total mentions
I agree with the other posters that the intro to NLP part is unnecessary. It reads like those recipe websites where they tell you their whole life story before the actual recipe. I get that it's good for SEO, but it's still annoying to read. Did you try other solutions like ULMFiT [1]? Seems like the exact use case for that. Although it might be overkill for just 4 categories. [1] https://arxiv.org/abs/1801.06146
These guys have done what looks like an impressive amount of work... but I was disappointed to see there's no mention of generative language modeling at all and only a very brief mention of key advances in image generation over the past few years. Examples of generative language modeling ignored by this paper (not even mentioned): * Unsupervised Transformer - https://blog.openai.com/language-unsupervised/ * ELMo - https://arxiv.org/abs/1802.05365 * ULMFit - https://arxiv.org/abs/1801.06146 Examples of generative image modeling ignored by this paper (some are mentioned only in passing): * Glow - https://blog.openai.com/glow/ * RealNVP - https://arxiv.org/abs/1605.08803 * NICE - https://arxiv.org/abs/1410.8516 * Pixel CNN - https://arxiv.org/abs/1606.05328 (and its cousin PixelRNN - https://arxiv.org/abs/1601.06759)
There is definitely a feedback loop at play in the NLP community. Since most NLP research is based on English, most applications that build off that research are also tailored to English. Our approach to classifying feedback in other languages is to simply use Google Translate's API and then proceed to classify the English translation. However, you are right in that many aspects of the base feedback are lost in this approach, and our accuracy is lower than if we had a language specific model for each language. That being said, there is a promising new research paper from fast.ai (https://arxiv.org/abs/1801.06146) that speaks to using Wikipedia data to create a language model for a specific language, which can then be trained specifically on the task you are trying to solve. If this is as effective as the authors state, then NLP could see huge improvements non-English languages where there is already a large set of Wikipedia data.
Yes, recent work on transfer learning https://arxiv.org/abs/1801.06146 " Fine-tuned Language Models for Text Classification" suggests that training a lstm on a large corpora for language modeling, then using it as sentence encoder after tuning it in specific dataset gives a considerable perfomance. This is covered in fast.ai 's new course 1.
This is fabulous, great work, with broad applicability. Train this transformer model on a good amount of text (or grab a pretrained model), and then, with minimal fuss and very little tweaking, you can repurpose it to obtain state-of-the-art (or near state-of-the-art) results in a wide range of tasks, from document classification to textual entailment to semantic similarity. This stands in contrast to prior approaches that involve much more tweaking and/or careful discriminative finetuning of the pretrained model, such as as Jeremy Howard and Sebastian Ruder's also-impressive ULMFit.[a] The main downside to this new approach is that pretraining takes a long time. Anyone working on ML/DL/AI with text should take a look at this, right now. UPDATE: See Jeremy Howard's comment here: https://news.ycombinator.com/item?id=17288320 [a] https://arxiv.org/abs/1801.06146
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.