Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, Aäron van den Oord
2020 · arXiv
Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
This number is not credible. Are we done with ImageNet?[1] points out that the best ImageNet models that currently exist surpass the accuracy of the original labels they are trained on. Expecting to reduce error by a factor 10 on unseen data points is incoherent. [1] https://arxiv.org/abs/2006.07159
Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on ImageNet really mean?) [4, 5, 6]. For (a), maybe surprisingly the answer is mostly yes! Better ImageNet accuracy generally corresponds to better out of distribution accuracy. For (b), it turns out that the ImageNet dataset is full of contradictions---many images have multiple ImageNet-relevant objects, and often are ambiguously or mis-labeled, etc---so it's hard to disentangle progress in…
identifying objects vs. models overfitting to the quirks of the benchmark. [1] ObjectNet: https://objectnet.dev / associated paper [2] ImageNet-v2: https://arxiv.org/abs/1902.10811 [3] An Unbiased Lookat Dataset Bias: https://people.csail.mit.edu/torralba/publications/datasets_... (pre-AlexNet!) [4] From ImageNet to Image Classification: https://arxiv.org/abs/2005.11295 [5] Are we done with ImageNet? https://arxiv.org/abs/2006.07159 [6] Evaluating Machine Accuracy on ImageNet: http://proceedings.mlr.press/v119/shankar20c.html
Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly collected the same way. So you have issues of both internal and external validity. In practice, at least for CNN classifiers on ImageNet etc, it seems to be a pretty minor issue overall. Some links: https://arxiv.org/abs/1711.11561 https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811 http://gradientscience.org/data_rep_bias/ http://gradientscience.org/data_rep_bias.pdf https://arxiv.org/abs/1905.10498 https://arxiv.org/abs/2002.02559 https://arxiv.org/abs/2006.07159 https://arxiv.org/abs/1805.08974 https://arxiv.org/abs/1902.10178 https://arxiv.org/abs/1912.11370#google
This article doesn't make any major factual errors, but it's implicating that this is some sort of surprise, and that it's a major impediment, neither or which hold up. While training on noisy datasets is an interesting problem for many practical use-cases, a few percent label errors in these classic basic datasets (MNIST, CIFAR, ImageNet) really doesn't matter that much. The first two are basically toys to check that your system works at all at this point, and even on ImageNet the best models are now more accurate than the original labels. However, per [1], progress on current ImageNet still correlates with true accuracy. This is in large because we move to harder tests…
when the easy ones stop working. In part it's also good, because training with label noise forces the use of better-generalizing solutions. The current SOTA is Meta Pseudo Labels[2], which is a particularly clever trick that never even directly exposes the final model to the training data. [1] Are we done with ImageNet? — https://arxiv.org/abs/2006.07159 [2] Meta Pseudo Labels — https://arxiv.org/abs/2003.10580
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.