Skip to content
The methodOpen data
Back to the collection

Intriguing properties of neural networks

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever

Show all 7 authors

Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus

2013 · arXiv

Machine learningPsychologyComputer vision
Read the paper
HN score
1
Mentions
21
Readers
20
Months
18
What’s in a score?

From the abstract

Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

20 selected excerpts · 21 total mentions

sotte
Recommendation · rule
If you're interested in the generalization ability of neural networks I can recommend the following paper: "Intriguing properties of neural networks" http://arxiv.org/pdf/1312.6199v4.pdf TLDR: The authors create adversarial examples, i.e., they slightly modify the original images, which look exactly the same to humans but neural networks can't classify the images correctly anymore. What does that imply on the ability to generalize? :) On a more general note: NN are always treated as something magic. I think a "sober view" is that NN are a special way to parameterize non-liner functions. This is neither good nor bad but it's easy to see when you look at, for example, a 2 layer NN: $f(x) = w_2^T \sigma(W_1 \sigma (W_0 x))$
Original comment A visual proof that neural nets can compute any function
YeGoblynQueenne
Neutral mention · rule
With the exception of counterfactual regret minimisation, which has nothing to do with deep learning, all those are applications of existing approaches that have become possible because of the increased expenditure of resources- processing power and data. There have been no theoretical advances, no algorithmic advances to speak of, no new knowledge learned in the last 20 years of reserach in deep learning. All that has been achieved is new SOTA on old benchmarks [1]. To make an analogy it's as if we could sort ever larger lists because we keep sorting them with ever bigger computers... but the only sorting algorithm we know is bubblesort. You bring up large language…
Read the full comment
models. Language models are about as old as computational linguistics [6]. Large language models are... large. There's nothing that's new about them. "Attention" is an anthropomorphically misnamed engineering tweak of the kind that was rightly lambasted by Drew McDermot fourty years ago [7]. Self-play and reinforcement Learning are now more than 60 years old [8]. As to "few shot" learning, that is said for a language model previously trained on a copy of the entire web! In what sense is that "few shot"? It is truly shocking to hear researchers make such obviously false claims with a straight face. And that they are rewarded with publication of their work is nothing less than a scandal. Deep learning research is going nowhere. It is stuck in a rut. It is dead. It has given up the spirit. It is an ex-research field. This research field is a goner. It is dead, jim. And I'm sure it will be decades before people start to say it out loud (the current generation of reserachers will leave the dirty job to their students), but that doesn't change a thing. ______________ [1] Except in identifying new weaknesses of deep learning systems, as for example Overinterpretation in the paper I cite above, adversarial examples [2], shortcut learning [3], "the elephant in the room" [4], being "right for the wrong reasons" [5] etc etc etc etc. [2] https://arxiv.org/abs/1312.6199 [3] https://www.nature.com/articles/s42256-020-00257-z [4] https://arxiv.org/abs/1808.03305 [5] https://aclanthology.org/P19-1334.pdf Which of course are very welcome. The only way to address the limitations of current approaches is to become aware of them and try to understand them. Unfortunately, it is only a tiny, tiny minority in the field who does that, while the majority is happy to spout nonsense like "[a billion plus] few shot" learning etc. [6] https://www.semanticscholar.org/paper/Speech-and-language-pr... [7] https://www.semanticscholar.org/paper/Artificial-intelligenc... [8] Arhtur Samuel's checkers player beat a human champion in 1961. Donald Michie build MENACE, a reinforcement learning algorithm implemented in matchboxes in 1960: https://rodneybrooks.com/forai-machine-learning-explained/ Ctrl+F for "Machine learning started with games" to find the relevant section.
Original comment Deep learning can’t be trusted, brain modelling pioneer says
sennight
Neutral mention · rule
It isn't "science fiction" though, we have a mountain of examples that demonstrate how probabilistic AI fails when the training dataset doesn't perfectly align with the eventual application. It isn't restricted to AI either, there have been numerous studies showing that cross-race identification to be garbage data. <Insert other race> all look the same: https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.898 This bus is an ostrich: https://arxiv.org/abs/1312.6199 Racist autofocus: https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio... Google thinks black people look like gorillas: https://www.wired.com/story/when-it-comes-to-gorillas-google... And here is an argument to look forward to: should the training dataset be build to represent the general population, or the specific subgroup that the software would be most often encountering? Because that FBI UCR...
Original comment Super Resolution
LoSboccacc
Neutral mention · rule
the Uber incident investigation showed how a continuous input generated wildly varying output in practice. in theory the existence of adversarial images and adversarial training has been studied for a while, but the state of the art can still produce singularities around continuous inputs. start here: https://arxiv.org/pdf/1312.6199.pdf
Original comment Autonomous DeLorean drives sideways to move forward
bagnalla
Neutral mention · rule
Verification of machine learning models, especially w.r.t. robustness to so-called "adversarial examples" [1], is a major research topic at the moment. There are certainly people worrying about it in the machine learning community, but it appears to be a very difficult problem. [1] https://arxiv.org/abs/1312.6199
Original comment Interview with Xavier Leroy
pinouchon
Neutral mention · rule
Hi. Thanks for the great article. Can you elaborate on what you mean by "directions in activation space" ? If I understand it right: You take a few neurons in a layer, and you follow some linear combination of their weights; you are then walking along a random direction. You take a single neuron and walk along its weights; you walk along this neuron's direction. Is this correct? Also another comment: Szegedy et al [9] (https://arxiv.org/pdf/1312.6199.pdf) has the following abstract: > First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests…
Read the full comment
that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks. This is one of the predictions of the following paper, even though I think it came later: Opening the Black Box of Deep Neural Networks via Information (https://arxiv.org/abs/1703.00810). Here: https://youtu.be/FSfN2K3tnJU?t=1h19m23s, Tishby says: "One of the consequences of this theory is that single neurons in a layer don't tell us much." Also, If you extend this idea to training examples, you get Mixup (https://arxiv.org/pdf/1710.09412.pdf)
Original comment Feature Visualization: How neural nets build up their understanding of images
haraldurt
Neutral mention · rule
Not necessarily. Adversarial examples have been shown to, for instance, be transferable across different networks with different hyperparameters (e.g., number of layers) trained on disjoint subsets of a training set [0, section 4.2]. There are more references from the paper linked by the OP. [0] https://arxiv.org/abs/1312.6199
Original comment How Adversarial Attacks Work
yorwba
Neutral mention · rule
Actually, no [1]. From the abstract: We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input. [1] https://arxiv.org/abs/1312.6199 EDIT: Also interesting: "Universal Adversarial Perturbations" https://arxiv.org/abs/1610.08401
Original comment AI Model Fundamentally Cracks Captchas, Scientists Say
moyix
Neutral mention · rule
Yes: > Adversarial training seeks to improve the generalization of a model when presented with adversarial examples at test time by proactively generating adversarial examples as part of the training procedure. This idea was first introduced by Szegedy et al. [SZS13] but was not yet practical because of the high computation cost of generating adversarial examples. Goodfellow et al. showed how to generate adversarial examples inexpensively with the fast gradient sign method and made it computationally efficient to generate large batches of adversarial examples during the training process [GSS14]. The model is then trained to assign the same label to the adversarial example…
Read the full comment
as to the original example—for example, we might take a picture of a cat, and adversarially perturb it to fool the model into thinking it is a vulture, then tell the model it should learn that this picture is still a cat. An open-source implementation of adversarial training is available in the cleverhans library and its use illustrated in the following tutorial. http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-... https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1412.6572
Original comment A Guide to Synthesizing Adversarial Examples
mholt
Neutral mention · rule
This is called "adversarial training" and is ineffective because of transferability. Also the space of adversarial examples is thought to be large (especially as dimensionality increases), making robust adversarial training intractable in practice. See https://arxiv.org/abs/1312.6199 and https://arxiv.org/abs/1510.05328
Original comment Optical illusions that flummox computers
BickNowstrom
Neutral mention · rule
Ensembles are also vulnerable to adversarial images. From one of the canonical papers on adversarial images: > In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input. > ... suggest that adversarial examples are somewhat universal and not just the results of overfitting to a particular model or to the specific selection of the training set https://arxiv.org/abs/1312.6199 More recent research shows that one can make adversarial images which fool ensembles. Once you have these images there is a very…
Read the full comment
high chance it will also fool any individual architecture you pass it on to next. https://twitter.com/goodfellow_ian/status/790330213963923456 These attacks show the inherent flaws of ML models (not just convnets, as per http://karpathy.github.io/2015/03/30/breaking-convnets/ ), and as far as I know, they are not trivial to work around. Research on adversarial images is still very active, in part because the military would like to rely on deep learning, and does not like to mistake a tank for a puppy (and vice versa).
Original comment Real and Stealthy Attacks on State-Of-the-Art Face Recognition [pdf]
obastani
Neutral mention · rule
It's not at all the same -- self-driving cars can risk the lives not only of the passengers, but of innocent bystanders. Several commenters have pointed out problems such as rain, snow, darkness, etc., but there are more fundamental problems. For example, self-driving cars based on machine learning may be susceptible to adversarial attacks that cause the car to behave unpredictably [1,2]. How will self-driving cars know how to react when things go wrong, e.g., the stop light is broken? What about construction? I'm not saying humans react perfectly in these situations, but self-driving cars are rule-based, and have zero ability to adapt to unanticipated situations. I'm not…
Read the full comment
arguing that Tesla shouldn't push forward. However, I believe that so far, Tesla has demonstrated a lack of concern for user safety. Companies like Google, Mercedes-Benz, etc. have tech at least as good as Tesla's, but have been much more cautious about deploying it. [1] https://arxiv.org/abs/1606.04435 [2] https://arxiv.org/abs/1312.6199
Original comment All Tesla Cars Being Produced Now Have Full Self-Driving Hardware
jkldotio
Neutral mention · rule
1. If that's the case perhaps another kind of blurring? "Intriguing properties of neural networks" (https://arxiv.org/pdf/1312.6199.pdf page 6) has examples where you get radically different classifications that I don't think would occur naturally or survive a blur with some random element, let alone two moving cameras and a sequence of images. As the title says it's an intriguing property, not necessarily a huge problem. 2. I honestly can't think of a situation where this could occur. It's the equivalent of kids shining lasers into the eyes of airline pilots, but the kids need a PhD in deep learning and specialised equipment to be able to do it. A hacker doing some…
Read the full comment
update to the software via a network sounds much more plausible than attacking the system through its vision while it's traveling. 3. This is the real point in the end I guess, this Google presentation (https://www.youtube.com/watch?v=tiwVMrTLUWg) shows that the first autonomous cars to be sold will be very sophisticated with multiple systems and a lot of traditional software engineering. Hopefully LIDAR costs will come down.
Original comment End to End Learning for Self-Driving Cars [pdf]
dllu
Neutral mention · rule
Failure to generalize is not always caused by overfitting. Even if there is no overfitting, deep neural networks seem to learn a surprisingly discontinuous function. Consequently, in rare cases, they can misclassify things with great confidence. [0] From the cited paper, > [Experimental results] suggest that adversarial examples are somewhat universal and not just the results of overfitting to a particular model or to the specific selection of the training set. Anyway, Monte Carlo Tree Search is bad at losing positions. In general, you want to delay the impending catastrophe as long as possible instead of making stupid moves that make your position worse and worse.…
Read the full comment
However, MCTS uses random rollouts to the end of the game, which sometimes make it difficult to ascertain if the inevitable doom is near or far. Also, MCTS converges very, very slowly and is likely to miss a unique, single winning continuation. I think it is probably a combination of both AlphaGo's value network failing to realize the good position of Lee Sedol after his brilliant play, and the MCTS failing to spot the unique winning sequence for Lee, that caused it to make the mistake. But we should probably wait for official analysis from the Deepmind team to see what exactly went wrong. [0] "Intriguing properties of neural networks" http://arxiv.org/pdf/1312.6199.pdf
Original comment Lee Sedol Beats AlphaGo in Game 4
Houshalter
Neutral mention · rule
To be clear I meant this paper (http://arxiv.org/abs/1312.6199) as the original paper for adversarial images. I think they did try transferring them between very different NNs: >In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.
Original comment Images that fool computer vision raise security concerns
yummyfajitas
Neutral mention · rule
Algorithmic biases often reflect human biases I've been doing this for a while and I've literally never met a human who told an algorithm to overweight x[23] ("good looking"), x[48] ("is white") and x[873] ("is wealthy"), for x a 1,100-dimensional feature vector. Algorithms do have biases, but they are almost always orthogonal to the human ones. Witness, for example, all the recent "we can fool deep learning image recognition systems" papers. http://arxiv.org/abs/1412.1897 http://arxiv.org/abs/1312.6199 At this point I'm 90% sure you are a layperson who's never actually programmed such a system.
Original comment Life Lessons from Machine Learning
colah3
Neutral mention · rule
You make some pretty strong attacks on the state of the art approach to computer vision. Could you please provide citations? As someone working on neural networks, although not really on computer vision, I do not believe several of the things you said are true. > > Pornographic images, for instance, are frequently identified not by the presence of particular body parts or structural features, but by the dominance of certain colors in the images > Neural networks This is definitely not what state of the art neural networks for computer vision do, although I'm sure there are many models out there that do function in that manner. We know that neural networks learn to detect…
Read the full comment
higher level features. For example, see the excellent visualizations in Matthew Zeiler and Rob Fergus' Visualizing and Understanding Convolutional Networks paper. (http://arxiv.org/pdf/1311.2901.pdf) > Neural networks are unsuccessful because they are not detecting anything. They're just making statistical correlations that happen to be right for the specific set of training data they have received. Are you saying that the correlations they learn aren't true on non-training data? If so, that is false. Any competent person in ML tests on different data than they train. Well trained neural networks generalize to new data very well. That's why we use them. > There's no robustness whatsoever. Artificial gullibility. Are you referring to the adversarial counterexamples section of Szegedy et al's Intriguing properties of neural networks paper? (http://arxiv.org/abs/1312.6199) This paper shows that one can craft noise which will fool a neural network. But it isn't noise that is found in real life. It's created by an optimization process which is basically looking for ways to fool the network. Basically, it discovers an optical illusion for the network. (From the manifold perspective, it moves us in a direction perpendicular to the manifold.)
Original comment A.I. Has Grown Up and Left Home
AndrewOMartin
Neutral mention · rule
Don't forget "Intriguing properties of neural networks", otherwise known as "Does Deep Learning have deep flaws?". In sum, you can teach a network to say "that's a dog" when presented with a picture of a dog, but you'll also be able to (1) find an imperceptibly modified version of the image where it'll say "that's garbage" and (2) intelligently generate an image of noise that also gets recognised as a dog. (1) http://arxiv.org/abs/1312.6199 (2) http://www.newscientist.com/article/dn26691-optical-illusion...
Original comment Deep Learning Reading List
abeppu
Neutral mention · rule
(a) The document linked by the OP is to a blog post that discusses and extensively quotes another blog post [1] which in turn discusses an actual paper [2]. Naturally the paper is where the good stuff is. (b) Both blog posts somewhat understate the problem. The adversarial examples given in the original paper aren't just classified differently than their parent image -- they're created to receive a specific classification. In the figure 5 of the arxiv version, for example, they show clear images of a school bus, temple, praying mantis, dog, etc, which all received the label "ostrich, Struthio camelus". (c) The blog post at [1] wonders whether humans have similar…
Read the full comment
adversarial inputs. Of course it's possible that we might, but I suspect that we have an easier time than these networks in part because: (i) We often get labeled data on a stream of 'perturbed' related inputs by observing objects in time. If I see a white dog in real life, I don't get just a single image of it. I get a series of overlapping 'images' over a period of time, during which time it may move, I may move, the lighting may change, etc. So in a sense, human experience already includes the some of the perturbations that ML techniques have to introduce manually to become more robust. (ii) We also get to take actions to get more/better perceptual data. If you see something interesting or confusing or just novel, you choose to focus on it, or get a better view because of that interestingness or novelty. The original paper talks about the adversarial examples as being in pockets of low probability. If humans encounter these pockets only rarely, it's because when we see something weird, we want to examine it, after which that particular pocket has higher probability. [1] http://www.i-programmer.info/news/105-artificial-intelligenc... [2] http://arxiv.org/abs/1312.6199 or http://cs.nyu.edu/~zaremba/docs/understanding.pdf
Original comment The Flaw Lurking in Every Deep Neural Net
ot
Neutral mention · rule
This is a very interesting perspective on neural networks, is it novel? I've never seen a geometric interpretation of NNs. In particular this: > (Apparently determining if knots are trivial is NP. This doesn’t bode well for neural networks.) Is there any theoretical research on obstructions to NN learning? Not that it would change much the practice (for instance, MLE learning of gaussian mixtures is NP-hard, but everybody does it anyway), but it could shed some light on the geometry of the "hard" instances. EDIT: for example, to get a sense of how ill-conditioned is deep NN learning, a recent paper [1] shows that if we feed to an object classifying NN an image with crafted but unnoticeable perturbations the predictions change completely. [1] http://arxiv.org/abs/1312.6199
Original comment Neural Networks, Manifolds, and Topology

Keep following the thread.

More in Machine learning

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score