Recent research has revealed that the output of Deep Neural Networks (DNN) can be easily altered by adding relatively small perturbations to the input vector. In this paper, we analyze an attack in an extremely limited scenario where only one pixel can be modified. For that we propose a novel method for generating one-pixel adversarial perturbations based on differential evolution (DE). It requires less adversarial information (a black-box attack) and can fool more types of networks due to the inherent features of DE. The results show that 67.97% of the natural images in Kaggle CIFAR-10 test dataset and 16.04% of the ImageNet (ILSVRC 2012) test images can be perturbed to at least one target class by modifying just one pixel with 74.03% and 22.91% confidence on average. We also show the same vulnerability on the original CIFAR-10 dataset. Thus, the proposed attack explores a different take on adversarial machine learning in an extreme limited scenario, showing that current DNNs are also vulnerable to such low dimension attacks. Besides, we also illustrate an important application of DE (or broadly speaking, evolutionary computation) in the domain of adversarial machine learning: creating tools that can effectively generate low-cost adversarial attacks against neural networks for evaluating robustness.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
On non-linearity of software systems. Correct me if I’m wrong but I’m assuming you mean it in a sense of being probabilistic and not easily predictable. Which is an interesting point because a lot of modern algorithms are like that. Let’s take neural networks. Being able to explain what the model is doing and how to make sure it’s correct is a whole area of research. For example, the results of a convolutional net can change drastically by altering a single pixel in an image[1].
We also faced with a problem of explaining a machine learning model that was much simpler than a neural net. It is a global problem and unfortunately I don’t have any solution of the top of my head.
[1] https://arxiv.org/abs/1710.08864
Superseding us in what way? The machine is already better at chess and Go, clearly not at classifying images[1], but the issue here is that the author argues that there is no such thing as a general intelligence. It's all situational.
On a separate note, does it even matter? Some people argue it's important so that we can put AI safe-guards in place, to protect mankind. But either:
A) Intelligence explosion won't happen and those safe-guards are not needed.
B) Intelligence explosion happens and the supersmart AI will easily find its way around feeble safe-guards put in place by the inferior humans.
While the end results are not the same; in either case the safe-guards are useless, though Asimov sure made a good story about them.
[1] https://arxiv.org/abs/1710.08864
> He knew exactly what to say to the responding officer to cover his ass
Replacing human judgment with algorithms is a terrible idea for many reasons, but this might be the worst. Algorithms are deterministic, and thus they can be manipulated. "SEO" services exist in spite of Google's efforts to protect their ranking algorithm. Does anybody seriously believe their local government will be more successful than Google at preventing people (i.e. police, prosecutors, etc) from "optimizing" their input to the algorithm?
edit - justification for the claim that a bail algorithm would be easy to manipulate:
The recent single-pixel attack[1] against image classifiers. https://arxiv.org/abs/1710.08864
Google's failure to recognize obviously inapropriate videos in their "youtube kids" app ("elsagate") https://medium.com/@jamesbridle/something-is-wrong-on-the-in... (a good summary & discussion https://www.youtube.com/watch?v=LKp2gikIkD8 )
Extensive discussion:
https://news.ycombinator.com/item?id=15577885
Also:
Paper identifier
'1710.08864.pdfhttps:/arxiv.org'
not recognized
There appears to be a valid arXiv paper identifier
within '1710.08864.pdfhttps:/arxiv.org'. Perhaps
the link you followed was intended as a link to
arXiv:1710.08864.
Links:
* https://arxiv.org/abs/1710.08864
* https://arxiv.org/abs/1710.08864.pdf
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.