Deep CNNs are known to exhibit the following peculiarity: on the one hand they generalize extremely well to a test set, while on the other hand they are extremely sensitive to so-called adversarial perturbations. The extreme sensitivity of high performance CNNs to adversarial examples casts serious doubt that these networks are learning high level abstractions in the dataset. We are concerned with the following question: How can a deep CNN that does not learn any high level semantics of the dataset manage to generalize so well? The goal of this article is to measure the tendency of CNNs to learn surface statistical regularities of the dataset. To this end, we use Fourier filtering to construct datasets which share the exact same high level abstractions but exhibit qualitatively different surface statistical regularities. For the SVHN and CIFAR-10 datasets, we present two Fourier filtered variants: a low frequency variant and a randomly filtered variant. Each of the Fourier filtering schemes is tuned to preserve the recognizability of the objects. Our main finding is that CNNs exhibit a tendency to latch onto the Fourier image statistics of the training dataset, sometimes exhibiting up to a 28% generalization gap across the various test sets. Moreover, we observe that significantly increasing the depth of a network has a very marginal impact on closing the aforementioned generalization gap. Thus we provide quantitative evidence supporting the hypothesis that deep CNNs tend to learn surface statistical regularities in the dataset rather than higher-level abstract concepts.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly collected the same way. So you have issues of both internal and external validity.
In practice, at least for CNN classifiers on ImageNet etc, it seems to be a pretty minor issue overall. Some links: https://arxiv.org/abs/1711.11561 https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811 http://gradientscience.org/data_rep_bias/ http://gradientscience.org/data_rep_bias.pdf https://arxiv.org/abs/1905.10498 https://arxiv.org/abs/2002.02559 https://arxiv.org/abs/2006.07159 https://arxiv.org/abs/1805.08974 https://arxiv.org/abs/1902.10178 https://arxiv.org/abs/1912.11370#google
In light of [0], there is both theoretical and experimental reason to believe that CNNs are mostly picking up fourier decomposition.
[0] https://arxiv.org/pdf/1711.11561.pdf
The AI winter happens when outside hype and investment money overshoots what can be delivered in the timescale needed for good ROI. Disappointments are followed with periods of underinvestment.
Neural Networks are already decades old invention. Underlying all this new boom is the same basic architecture, alternating layers of affine transformations and nonlinear transformations trained with backpropagation and gradient descent.
What made the field suddenly explode was GPGPU's and bunch of tweaks that helped to solve vanishing/exploding gradients problems (started with RNN training layers and now with skip connections and handful of other techniques that make deep networks…
Read the full comment
possible) combined with better regularization etc. I'm not trying to downplay the innovation, but from the larger point of view they are technical tweaks.
There is limit to where tweaking/scaling the current techniques can go. There needs to be leaps.
Geoff Hinton made the case better in his "What is wrong with convolutional neural nets?" https://www.youtube.com/watch?v=Jv1VDdI4vy4 see also https://arxiv.org/abs/1711.11561
from Jason Jo, Yoshua Bengio. The same "what's wrong with" applies to the field in general.
Hinton's capsule networks are attempt to take another leap forward. Just like with his earlier work from 80's that culminated around 2006, it may take years and years of slow work to get there. Hinton suggests that there needs to be unsupervised learning revolution that comes up with something else than backprobagation. https://www.axios.com/artificial-intelligence-pioneer-says-w...
This seems like it could dramatically worsen overfitting-like effects for algorithms like CNNs for image processing, where surface statistics of the available data set seem to be more responsible for the learned model than any type of “semantic” understanding.
If you prespecify what data augmentation you would do, like preregistering the details of a clinical trial, you’ll be less susceptible to a spurious result from this.
It seems like especially things like color distribution manipulation would have a potentially very adverse effect that counters any gains from clamping the supervised learning to be “robust” to that color variation.
I’m thinking in the spirit of: < https://arxiv.org/abs/1711.11561 >.
One of the cited papers "Measuring the tendency of CNNs to Learn Surface Statistical Regularities" is also a great insight into this phenomena [0].
> Our main finding is that CNNs exhibit a tendency to latch onto the Fourier image statistics of the training dataset, sometimes exhibiting up to a 28% generalization gap across the various test sets. Moreover, we observe that significantly increasing the depth of a network has a very marginal impact on closing the aforementioned generalization gap. Thus we provide quantitative evidence supporting the hypothesis that deep CNNs tend to learn surface statistical regularities in the dataset rather than higher-level abstract concepts.
[0] https://arxiv.org/abs/1711.11561
As one example of how this can be useful, Jo and Bengio recently used Fourier filtering to measure the susceptibility of neural networks to adversarial examples. By changing the statistics of the images in a principled manner, they confirmed that even networks that generalize well are learning mostly surface-level statistics. E.g. an image of a car is more likely to also have asphalt and building colors than greenery. The networks overweight these kinds of features, and that turns out to be good enough to get high scores on many data sets. Using Fourier filtering, they were able to alter the images to generate arbitrarily different surface statistics while preserving how humans would perceive the images.
https://arxiv.org/abs/1711.11561 (https://news.ycombinator.com/item?id=16165126)
I would say training a CNN from scratch or even fine-tuning one takes a lot of domain knowledge and best practices which often are not standardised yet. Besides, we don't even know why they generalise in the first place! See: https://arxiv.org/abs/1611.03530, https://arxiv.org/abs/1711.11561
A better title for the article might have been,
“Why current deep learning is not general enough for AGI or some other families of structured problems.”
Viewed this way, it’s not a criticism of pragmatically using deep learning or experimenting with deep learning for some narrow tasks.
Rather, it expresses what aspects of current deep learning make it unsuitable for general transfer learning, hierarchical or causal inference, or many Bayesian techniques requiring greater use of priors.
Other comments have pointed out that there are plausible rebuttals in deep reinforcement learning and metalearning.
But the bigger thing to me is to be clear that the article is not a…
Read the full comment
criticism of deep learning engineering — applying deep learning to satisfy an explicit requirement, where the success criteria possibly has nothing at all to do with general intelligence nor with whether a certain approach can span some sufficiently large class of general models.
However, even if you just constrain your view to look just at so-called “pragmatic” deep learning — deep learning for concrete tasks — there are still a lot of unanswered questions about why things work, and whether or not an approach is learning semantic aspects of some true underlying structure (some latent variable space that captures the true data generating process) or if deep learning merely allows overfitting to particular populations of observation-space statistics.
This paper gives an example of exactly this issue for CNN models for image processing [0]. I’d argue that this is more of the kind of criticism relevant for task-oriented practitioners, whereas the OP link is some criticism more relevant for AGI or philosophy of statistics research at large.
[0]: < https://arxiv.org/pdf/1711.11561.pdf >
I don’t work in the “high-tech HR” space, but do have experience with embedding vector methods, joint-embeddings, and trying to solve generic information retrieval tasks, where the underlying data is not text or images, by using embedding methods.
Even in a domain like reverse image search, where this approach has been studied to death, there are big concerns about how much of the problem is solved because a vector similarity approach can approximate the real manifold distance of some underlying true structure vs. just having huge models overfit to a particular class of natural image statistics.
This problem is discussed in [0], which creates a lot of problems for people…
Read the full comment
who want to believe that some deep, internal layer of a neural net can capture semantically relevant features.
But a much bigger problem looms for trying to extend this idea to matching people to jobs. In that problem, you don’t even have the option of overfitting to population statistics because the population is constantly changing and the individuals in the population have an insanely high-dimensional set of internal unobserved variables, like their emotions, goals, current life or family priorities, interests, relative free time, sensitivity to stress, etc.
By comparison, the space of latent variables giving rise to observed natural photos is tiny.
Essentially, if you really wanted to take a scary, Orwellian, big data approach to quantifying a candidate’s degree of match to a job, you would need much more data on the conditional distribution of the observables (resume items, college degree, skills assessment, etc) when given information on the internal state (work ethic, introvert vs extravert, motivation for looking for a job, intelligence metrics, disposition, response to stress, etc) at a given moment of time.
This would let you model the posterior distribution of those hidden, internal characteristics of the applicant, and those characteristics could maybe be used to understand a holistic match to a certain employer-team-role situation.
But that’s a ludicrously high-dimensiomal problem that observable data like resumes or skills assessments does little to solve.
So overall, I probably share your opinion that this is an extremely shallow model.
It’s like learning a vector space model of spaghetti and a vector space model of walls and then claiming your model can predict what will stick when you throw spaghetti at the wall.
[0]: < https://arxiv.org/abs/1711.11561 >
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.