Nicolas Papernot, Patrick McDaniel, Ian Goodfellow
Somesh Jha, Z. Berkay Celik, Ananthram Swami
2016 · arXiv
Machine learning (ML) models, e.g., deep neural networks (DNNs), are vulnerable to adversarial examples: malicious inputs modified to yield erroneous model outputs, while appearing unmodified to human observers. Potential attacks include having malicious content like malware identified as legitimate or controlling vehicle behavior. Yet, all existing adversarial example attacks require knowledge of either the model internals or its training data. We introduce the first practical demonstration of an attacker controlling a remotely hosted DNN with no such knowledge. Indeed, the only capability of our black-box adversary is to observe labels given by the DNN to chosen inputs. Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN. We use the local substitute to craft adversarial examples, and find that they are misclassified by the targeted DNN. To perform a real-world and properly-blinded evaluation, we attack a DNN hosted by MetaMind, an online deep learning API. We find that their DNN misclassifies 84.24% of the adversarial examples crafted with our substitute. We demonstrate the general applicability of our strategy to many ML techniques by conducting the same attack against models hosted by Amazon and Google, using logistic regression substitutes. They yield adversarial examples misclassified by Amazon and Google at rates of 96.19% and 88.94%. We also find that this black-box attack strategy is capable of evading defense strategies previously found to make adversarial example crafting harder.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
6 selected excerpts · 6 total mentions
Nicholas Papernot [1] et al. has shown how this can be an attack on an unknown network provided you have access to inference results produced by the target network [2]. [1] https://www.papernot.fr/ [2] https://arxiv.org/abs/1602.02697
One method of creating adversarial examples in a "black box" setting is to create and train a local model as a stand-in for the actual model using the inputs and outputs of the actual model. [1] So, the answer is "no" but a qualified "no" since in practice this seems to work. The second part, being able to forward many images, is also a qualified "no". 1 - https://arxiv.org/abs/1602.02697
You don't need access to the network, you can just build another one. https://arxiv.org/abs/1602.02697 "Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN"
No, this paper by Papernot et al shows how to do blackbox attacks without knowledge of model internals. https://arxiv.org/abs/1602.02697
Interestingly, you can treat the NN as a black box (ie, not look at individual weights or even the architecture) and still derive adversarial cases: https://arxiv.org/abs/1602.02697
http://arxiv.org/abs/1602.02697 I don't work on reCAPTCHA but I imagine they could easily + significantly beef up image captcha by adding adversarial image examples that trip up would-be automators. Note that adversarial examples generalize across models. Meaning a good adversarial example will work against SVMs, convnets, etc just the same. If you're interested in the basics of how it works, Julia Evans has a great post: jvns.ca/blog/2015/12/24/how-to-trick-a-neural-network-into-thinking-a-panda-is-a-vulture The key is that deep neural nets are actually very (piecewise) linear and that makes them susceptible to adversarial training.
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Scott Aaronson
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.