Given a state-of-the-art deep neural network classifier, we show the existence of a universal (image-agnostic) and very small perturbation vector that causes natural images to be misclassified with high probability. We propose a systematic algorithm for computing universal perturbations, and show that state-of-the-art deep neural networks are highly vulnerable to such perturbations, albeit being quasi-imperceptible to the human eye. We further empirically analyze these universal perturbations and show, in particular, that they generalize very well across neural networks. The surprising existence of universal perturbations reveals important geometric correlations among the high-dimensional decision boundary of classifiers. It further outlines potential security breaches with the existence of single directions in the input space that adversaries can possibly exploit to break a classifier on most natural images.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
I am unsure what you mean-- do you mean with different training sets but the same testing set?
It's an interesting question; maybe the reason for (some) of these adversarial vulnerabilities is due to a handful of bad training examples.
You could formulate it as a search problem to see if there's particular images (or small groups of images) that are responsible for the adversarial vulnerabilities.
This might then indicate that some of these perturbations are really just taking advantage of the fact that neural nets tend to "memorize" some of the data, so we're not really exploiting some deep structural feature so much as just feeding the echo of an input that the net has…
Read the full comment
learned to automatically classify as, say, a computer/desk[0].
It would be a good project, but I don't have enough GPUs on hand to train scores of deep nets from scratch.
Assuming one were to bite the bullet, it might also be worth trying different data augmentation strategies.
Most of the time, we try to eke out additional performance/robustness by using the same sets of transformations (translation, rotation, cropping, rescaling, etc.), but if the net is vulnerable to adversarial examples because of something in the training set, then you might just be making sure that adversarial vulnerability is present everywhere in the image and at multiple scales.
On a related note, there's an interesting paper about universal adversarial perturbations, i.e. those that can be added to any image and thereby induce a misclassification with high probability[1].
This effect holds even across different models, so the same perturbation can cause a misclassification in different architectures.
------
0. Neural nets learn by some combination of abstraction and memorization.
If, for some reason, many members of a particular class are hard to generalize, then it's possible that they instead learn to identify some particular aspects of those classes (that are not usually present in other images) and have a disproportionate response when those features are present.
If such features are not obvious to human visual inspection, then we get misclassifications without insight into why they were misclassified.
1. https://arxiv.org/pdf/1610.08401.pdf
Speaking of similarity between CV and human visual systems; it was kinda striking to me just how psychedelic/"trippy" the "universal adversarial perturbations" (see page 5 in https://arxiv.org/pdf/1610.08401.pdf) look.
Indeed, the look of visual hallucination "form constants" is intimately related to the machinery in the visual cortex that detects edges/contours/surfaces: https://www.math.uh.edu/~dynamics/reprints/papers/nc.pdf and such visual hallucinations can be elicited without any drugs or physiological interventions/defects -- just via diffuse flickering light: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3182860/
Actually, no [1]. From the abstract:
We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.
[1] https://arxiv.org/abs/1312.6199
EDIT: Also interesting: "Universal Adversarial Perturbations" https://arxiv.org/abs/1610.08401
This is obviously data dependent. I suspect that the advantage of human is much higher in colloquial content vs written (esp. news) content. "Universal Adversarial Perturbations"[1] last year showed that you can easily generate reasonable (to human) perturbations to completely fool the state of the art DNNs for images. I suspect that the same is true for the current batch of NMTs as well. As a simple demo, I change the example Chinese a little (就要下班了。下班了再说吧。Only aux characters changes with the same meaning) and all NMTs failed spectacularly in different ways.
Google: It is necessary to get off work. To say it again.
OpenNMT: it's going to work. Go back to work again.
Baidu: It's going to work. After work.
[1] https://arxiv.org/abs/1610.08401
"Generative models are only vulnerable to adversarial examples that are actually unlikely in the data distribution. They do not have patterns or filters that can be added to ordinary images to cause wild misclassifications."
^ Actually, you are wrong on this. See this recent paper "Universal adversarial perturbations"
https://arxiv.org/pdf/1610.08401v1.pdf
This reminds me of those blinky goggles that made you see a bunch of trippy patterns (even though the blinkiness was completely diffuse!), here's the instructions (http://wealoneonearth.blogspot.com/2009/09/how-to-build-gogg...) and here's a proper scientific paper about this phenomenon: http://journals.plos.org/ploscompbiol/article?id=10.1371/jou... .
The interesting thing is that the patterns that recur (known as "form constants, https://en.wikipedia.org/wiki/Form_constant ) in these geometric visual hallucinations are pretty similar to those obtained by taking hallucinogenic drugs (like LSD). As one might expect, these "form constants" don't directly originate from the…
Read the full comment
flickering light nor the LSD molecule -- there's certainly no way to encode a description of those swirling geometric patterns in a small molecule or in diffuse light -- the information for those hallucinations has to reside in the brain somehow. Fascinatingly enough, these trippy geometric hallucinations are generated by the exact same brain structures and mechanisms that the visual system uses to process edges/contours, surfaces, and textures: http://www.math.utah.edu/~bresslof/publications/01-3.pdf ! Certain operational modes (that LSD or diffuse flickering light can help create) in the edge/contour/surface/texture-detection machinery -- once transformed to the visual field's coordinates -- look exactly like the form constants. These geometric visual hallucinations -- the imagery fed to a conscious mind whose visual system is temporarily malfunctioning -- have a deep correspondence with the normal mechanisms of vision, in much the same way that the "glitches" produced by feeding a H.264 decoder with a slightly corrupt video file let you immediately appreciate the motion-compensating machinery of the codec.
There is sublime beauty in seeing these "form constants" in images that look like geometric visual hallucinations appear in machine learning research. The universal adversarial perturbations (which, when added to an image, are said to cause misclassification) in page 6 of https://arxiv.org/pdf/1610.08401v1.pdf were striking to see, as they strongly resemble the same beautiful swirly colourful geometric visual hallucinations that humans experience when taking a hallucinogenic drug (or staring at diffuse flickery lights with the hallucination goggles).
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.