Do CIFAR-10 Classifiers Generalize to CIFAR-10?

No. I was referring to the "standard concentration bound" in that paper, which applies when you have separate validation and test sets. I think the argument can usually be improved by applying small-variance inequalities such as Bernstein's, to excess risk-like quantities such as l(f_hat(x), y) - l(f_ref(x), y), to show that accuracy difference / relative rank enjoys better guarantees. For ImageNet we can use the 01 loss and set f_ {ref} to a SoTA classifier which, while having its loss bounded away from 0, is "mostly similar" to most f_hat's, and thus leads to a small excess risk. The CIFAR experiments I mentioned were https://arxiv.org/pdf/1806.00451.pdf. It doesn't contain this argument (unfortunate wording) but appears to support it well.

I disagree: some "collective overfitting" happens when everyone evaluates on the same dataset (and often, the same test set). There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples between the train and test partitions causes systems to perform up to 10-15% worse. https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811 There's another neat bit of work involving MNIST. The original dataset (from the mid-90s) had 60,000 test examples, but the distributed versions that virtually everyone uses has only 10,000 test examples. Performance on these held-out examples is, unsurprisingly, a bit worse: https://arxiv.org/pdf/1905.10498.pdf

Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly collected the same way. So you have issues of both internal and external validity. In practice, at least for CNN classifiers on ImageNet etc, it seems to be a pretty minor issue overall. Some links: https://arxiv.org/abs/1711.11561 https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811 http://gradientscience.org/data_rep_bias/ http://gradientscience.org/data_rep_bias.pdf https://arxiv.org/abs/1905.10498 https://arxiv.org/abs/2002.02559 https://arxiv.org/abs/2006.07159 https://arxiv.org/abs/1805.08974 https://arxiv.org/abs/1902.10178 https://arxiv.org/abs/1912.11370#google

Yes, over-fitting is definitely something that happens. Here's a paper that demonstrates the problem by testing CIFAR classifiers on a new version of the input data: https://arxiv.org/abs/1806.00451 The flip side is that by having a common problem to tackle it's much easier to compare and contrast different results to figure out what really works. Many approaches that improve results on CIFAR don't scale to ImageNet, and many of the ImageNet papers don't scale to the even larger datasets that people are using now.

Many years of "human optimization" on the same old datasets must lead to severe overfitting, so these results should be taken with a grain of salt. Related research: https://arxiv.org/abs/1806.00451

Treating training datasets as dynamic components of ML systems, along with the corresponding tooling and infrastructure, is one of the most under-appreciated points in the field today. Part of the cause, I think, is due to stigma coming from the academic side that dataset collection is a low-level problem not worthy of serious algorithmic investment (good luck submitting a CVPR paper which improves on labeling speed or taxonomy management in your dataset warehouse). Datasets are considered a given, which also has the interesting side effect of massive hyperparameter and architecture overfitting, evidenced by the recent analysis of CIFAR-{10,100} [1]. In more applied and engineering settings though, it's good to see this area seeing a lot more investment, especially on the tooling side. [1] https://arxiv.org/abs/1806.00451

This paper is definitely quite interesting, but before this thread turns into a bunch of NN hate, here is another take: "Do CIFAR-10 Classifiers Generalize to CIFAR-10?" - https://arxiv.org/abs/1806.00451 They use the same procedure used to construct CIFAR-10 to construct a new test set and test a bunch of state of the art results on the new test set. They see a generalization gap, but the relative order of SoTA results remains (roughly) the same. So, yes, test set validation is an overestimate of real-world performance of these systems, but progress on test sets is indicative of progress in real-world settings. And to remember this in context, no-one is asking "do traditional computer vision systems understand images", because they were explicitly just looking at image statistics.