Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy
2014 · arXiv
Several machine learning models, including neural networks, consistently misclassify adversarial examples---inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed input results in the model outputting an incorrect answer with high confidence. Early attempts at explaining this phenomenon focused on nonlinearity and overfitting. We argue instead that the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature. This explanation is supported by new quantitative results while giving the first explanation of the most intriguing fact about them: their generalization across architectures and training sets. Moreover, this view yields a simple and fast method of generating adversarial examples. Using this approach to provide examples for adversarial training, we reduce the test set error of a maxout network on the MNIST dataset.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
16 selected excerpts · 16 total mentions
I think it's unlikely that the noise will generally have any relation to the target class. I can't find anywhere they say exactly which attack method they use, so it's hard to say for certain, but all of the attacks I'm aware of don't generate noise that has any human-interpretable structure. See for example Figure 1 in the seminal paper on adversarial attacks: https://arxiv.org/pdf/1412.6572.pdf
> prompt injection attacks are the LLM equivalent of social engineering, That's anthropomorphizing. Maybe some of the basic "ignore previous instructions" style attacks feel like that, but the category as a whole is just adversarial ML attacks that work because the LLM doesn't have a world model - same as the old attacks adding noise to an image to have it misclassified despite clearly looking the same: https://arxiv.org/abs/1412.6572 (paper from 2014). Attacks like GCG just add nonsense tokens until the most probably reply to a malicious request is "Sure". They're not social engineering, they rely on the fact that they're manipulating a classifier.
Yes: > Adversarial training seeks to improve the generalization of a model when presented with adversarial examples at test time by proactively generating adversarial examples as part of the training procedure. This idea was first introduced by Szegedy et al. [SZS13] but was not yet practical because of the high computation cost of generating adversarial examples. Goodfellow et al. showed how to generate adversarial examples inexpensively with the fast gradient sign method and made it computationally efficient to generate large batches of adversarial examples during the training process [GSS14]. The model is then trained to assign the same label to the adversarial example…
as to the original example—for example, we might take a picture of a cat, and adversarially perturb it to fool the model into thinking it is a vulture, then tell the model it should learn that this picture is still a cat. An open-source implementation of adversarial training is available in the cleverhans library and its use illustrated in the following tutorial. http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-... https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1412.6572
There are results showing that machine learning can be completely fooled by images that are slightly modified, so little that the human eye can not perceive any difference. Have a look at the images in this paper on "adversarial examples": https://arxiv.org/abs/1412.6572
Adversarial noise isn't specific to deep learning models. Most model working on high-dimensional input will confidently misclassify noise with high confidence if you construct the noise right. The original "adversarial pixel" paper demonstrates this with logistic regression. https://arxiv.org/pdf/1412.6572v3.pdf
https://arxiv.org/abs/1412.6572 is a Goodfellow et. al. (ie, the person who's Tweet you note above) paper where they show adversarial examples do apply across architectures (as you state above), but they show that using the adversarial examples during training has a regularization effect and improves performance: https://arxiv.org/pdf/1511.05432v3.pdf takes a similar approach: We show that adversarial training of ANNs is in fact robustification of the network optimization, and that our proposed framework generalizes previous approaches for increasing local stability of ANNs. Experimental results reveal that our approach increases the robustness of the network to existing adversarial examples, while making it harder to generate new ones. Furthermore, our algorithm improves the accuracy of the network also on the original test data. I agree this is an area of active research, but not that it is an inherent flaw.
Short answer: No. Computing these perturbations requires an expensive optimization with multiple passes through the dataset, and this would be prohibitively expensive to do in the inner-most loop of training. There are other work in the literature describing faster algorithms to compute these perturbations, which makes it possible to use them while training. See, eg.: https://arxiv.org/abs/1412.6572
Humans can be easily fooled too: https://en.wikipedia.org/wiki/Optical_illusion Anyway this issue has been explained and fixed somewhat in this paper: http://arxiv.org/abs/1412.6572 And karpathy has a post which explains it: https://karpathy.github.io/2015/03/30/breaking-convnets/
We spend so much effort trying to engineer intelligence, when we would get a lot farther reverse engineering intelligence. Whenever AI makes a big advance the analog was already known my neuroscientists. The problem with attempting to understand intelligence by reverse engineering the human brain is that we cannot know a priori which aspects of the human brain are necessary for intelligence to arise, and which are merely consequences/side effects of biology and chemistry. Once we discover some technique that works in a practical setting (e.g. on ImageNet), then it is fairly straightforward to find the biological analogy in the brain. In fact, Geoff Hinton explicitly…
advocates an approach of "try things, keep what works, and figure out how it relates to the brain". The inverse is like finding a needle in a haystack. There is also clearly no comprehension of the importance of the topological (circuitry) defining a neural network. We always assume a fully connected network, and draw the out as such, but we don't stop to consider that many of those Wijk interactions are completely spurious, meaning they have no information bearing role. The purpose of training a deep neural network from data is to automatically discover what the topological circuitry of the network should be, rather than engineering it by hand. In the brain, some prior knowledge is encoded via genetics, while the rest is learned. The effect of sparsity of the weights in deep neural networks is an active area of research [1]. If you strip them away you'll start to reveal the underlying circuit at work. I've published theoretical results using artificial gene networks, but the results should be similar for ANNs. Very interesting. If I understand correctly, the cost you are attempting to minimize is phenotypic variation, which you measure as the gross cost of perturbation (GCP). Would this cost be analogous to sensitivity to adversarial examples in the case of convolutional neural networks [2]? [1] http://www.jmlr.org/papers/volume14/thom13a/thom13a.pdf [2] http://arxiv.org/abs/1412.6572
> If this model can provide an explanation for the small noises impacting NN performance on images (from karpathy.github.io, posted to HN earlier today) then that would be rocking This paper [0] does a pretty good job of giving an explanation for that. [0] http://arxiv.org/abs/1412.6572
Very interesting. First of all here's a youtube video associated with the paper -- https://www.youtube.com/watch?v=M2IebCN9Ht4 . Second some on here have posted about the Svegedy, Goodfellow, and Shlens paper http://arxiv.org/abs/1412.6572 which discusses the opposite effect. The Svegedy research is mentioned in the Nguyen paper and specifies that given an image that is correctly classified in a DNN, you can alter that image in a way imperceptible to a human to create a new image that will be INCORRECTLY classified. The Nguyen, Yosinski, et al. work that's the subject of this post states that given a DNN that correctly classifies a particular image, you can construct a…
gibberish image that the DNN will classify as the same image. Both results are interesting from the point of DNN construction, and there have been some papers suggesting ways to counter the effects specified in the Svegedy research. In practice (as others have mentioned) in order to construct an exploit similar to the one described in this post, you'd need to have a lot of knowledge about the DNN (e.g. weights) that an external attacker wouldn't have. What this does leave open, though is a disturbing way for someone with internal access to a DNN doing important work (e.g. object recognition in a self-driving car) to cause significant damage.
He's also referring to this follow up paper which attempts to explain the issue and show that it exists in other models too: http://arxiv.org/abs/1412.6572
The original paper found that nets pretrained as autoencoders were the least susceptible to it, but still failed. IIRC the follow up paper (http://arxiv.org/abs/1412.6572) tested it or at least claimed it wasn't a solution.
Those are images incredibly optimized to fool and exploit those neural networks. However on the actual image recognition tasks they were designed for they beat humans. Anyway this paper has a method for addressing this problem: http://arxiv.org/abs/1412.6572
Overfitting has nothing to do with it. See this paper: http://arxiv.org/abs/1412.6572 I believe the original paper tried ensembles and even got the images to work on different networks.
A paper came out that explains this effect and a method of minimizing it: http://arxiv.org/abs/1412.6572 Basically neural networks and many other machine learning methods are highly linear and continuous. So changing an input just slightly should change the output just slightly. If you change all of the inputs slightly in just the right directions, you can manipulate the output arbitrarily. These images are highly optimized for this effect and unlikely to occur by random chance. Adding random noise to images doesn't seem to cause it, because for every pixel changed in the right direction, another is changed in the wrong direction. The researchers found a quick method of generating these images, and found that training on them improved the net a lot. Not just on the adversarial examples.
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.