Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza
Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio
2014 · arXiv
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
15 selected excerpts · 15 total mentions
It's quite remarkable how much the goal posts have shifted when it comes to what is impressive with AI/ML. Things like this are a good reminder. 10 years ago the GAN paper came out and everyone was excited how amazing the generated image quality was (https://arxiv.org/abs/1406.2661) The amount of progress we've made is mind boggling.
Yes and no. The thing is that Mamba is not perfect. There's no neural architecture to rule them all, if you will. I think the bigger issue is that we more act like there is and get on bandwagons. Let me give a clearer example from the past so we can see. The predecessor to DDPM (the work that kicked off the diffusion model era) was published in 2015[0], only a year after GANs[1]. Diffusion then showed promise but why did DDPM come out in 2020[2]? Because everyone was working on GANs. GANs produced far better images and diffusion (still is) was a lot more computationally intensive. Plus, all the people working on these diffusion models were in the same camp as those…
working on Normalizing Flows and other density based models, and fewer people are interested in understanding density estimation. So the entire problem is that the community hopped on a singular railroad for research direction. There was still working going on in that direction but it wasn't nearly getting the attention that GANs got. It's hard to know if things were blocked from publication because they weren't as good as GANs. I can say from personal experience I had a Flow publication blocked because reviewers were concerned with its quality compared to GANs (this was 2019/2020, this paper will never be published because now it is even hard to publish a GAN work). So yes and no because there is certainly railroading happening but there are also real critiques to Mamba. But what people often forget is that it is incredibly naive to compare new methods to existing methods on a direct one-to-one comparison. You're comparing something that has hundreds of hours to thousands of hours from a handful to a few dozen eyes against works with millions of hours and millions of eyes. Evaluation is just a really fucking hard thing to do but it is easy to just look at some benchmarks, even if they don't mean much. This is a fairly generalization notion though, so take the lesson to heart. But Mamba seems a bit different than our diffusion/GAN story, in that it is getting more attention than diffusion did in the 2016-2019. [0] https://arxiv.org/abs/1503.03585 [1] https://arxiv.org/abs/1406.2661 [2] https://arxiv.org/abs/2006.11239
This is a fine list, but it only covers a specific type of generative AI. Any set of resources about AI in general has to at least include the truly canonical Norvig & Russel textbook [1]. Probably also canonical are Goodfellow's Deep Learning [2], Koller & Friedman's PGMs [3], the Krizhevsky ImageNet paper [4], the original GAN [5], and arguably also the AlphaGo paper [6] and the Atari DQN paper [7]. [1] https://aima.cs.berkeley.edu/ [2] https://www.deeplearningbook.org/ [3] https://www.amazon.com/Probabilistic-Graphical-Models-Princi... [4] https://proceedings.neurips.cc/paper_files/paper/2012/file/c... [5] https://arxiv.org/abs/1406.2661 [6] https://www.nature.com/articles/nature16961 [7] https://www.nature.com/articles/nature14236
I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain) 2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal) 3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain) 4. "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks" (2016) -…
https://arxiv.org/abs/1511.06434 (University of Montreal) 5. "ImageNet Classification with Deep Convolutional Neural Networks" (2012) - https://papers.nips.cc/paper/4824-imagenet-classification-wi... (University of Toronto) 6. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2018) - https://arxiv.org/abs/1810.04805 (Google) 7. "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (2019) - https://arxiv.org/abs/1907.11692 (Facebook AI) 8. "ELMo: Deep contextualized word representations" (2018) - https://arxiv.org/abs/1802.05365 (Allen Institute for Artificial Intelligence) 9. "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" (2019) - https://arxiv.org/abs/1901.02860 (Google AI Language) 10. "XLNet: Generalized Autoregressive Pretraining for Language Understanding" (2019) - https://arxiv.org/abs/1906.08237 (Google AI Language) 11. T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (2020) - https://arxiv.org/abs/1910.10683 (Google Research) 12. "Language Models are Few-Shot Learners" (2021) - https://arxiv.org/abs/2005.14165 (OpenAI)
The arms race was first automated in 2014, when the GAN paper was published. That's what GANs do: half the network generates garbage, the other half identifies if it was garbage or real. https://arxiv.org/abs/1406.2661
With regards to his claim that he described a GAN before Ian Goodfellow did, it certainly seems like that is the case. The problem is that, in his original Artificial Curiosity paper he talks so specifically about world-simulating reinforcement learning. He even uses the terms "World Model" and "Controller" instead of the more general "Discriminator" and "Generator." In this paper, he says the GAN is a more specific application of his Artificial Curiosity paper, but I see it the other way around. Goodfellow should have cited the Artificial Curiosity paper though, as training two networks at odds with each other in parallel is explicitly mentioned by Schmindhuber in his…
original paper. It also doesn't hurt that Bengio and others came out with research when you could actually run the damn thing. Schmindhuber's Artificial Curiosity paper doesn't list a single experimental result. Schmindhuber perhaps should have shared in the Turing award, but at some point you have to cut it off. We stand on the shoulders of giants, and so do those giants. At some point, a giant is getting left out of the award. [1] Schmindhuber's original Artificial Curiosity paper: https://people.idsia.ch/~juergen/FKI-126-90ocr.pdf [2] Ian Goodfellow's original GAN paper: https://arxiv.org/abs/1406.2661
The work of Art is the act of putting it up for auction on Christie's. That's what makes it a conceptual piece that is being discussed. People often mistake art for a form of labor production. They expect that labor has to have been performed to create value. But that all changed in the early 20th century. I really feel bad for Dr. Beef, this must really hurt. But according to the twitter thread ( https://twitter.com/DrBeef_/status/1055285640420483073 ) they didn't use the pre-trained models. In any case pre-trained models are labor, not end product. They simply trained them from scratch. Possibly on the same input set. Even if his license was more strict, they still…
could have just rewritten the same code and it would have escaped the licensing. Should the original GAN paper ( https://arxiv.org/abs/1406.2661 ? ) be given artistic credit here ? What is authorship in this context ? There is a whole chain of people furthering things and doing labor and making decisions. Yet nobody has even mentioned the original artists whose works were used to train the model. Would it be different if they were living artists ? In what way is a GAN essentially different than photoshop copy and paste of elements ?
> but isn't that the whole point of GANs? not quite, but youre on the right path. think about it this way: you (the generative model) are trying to predict a unit gaussian, which is just a fancy way to say bell curve. you get +1 if you predict a number in this distribution (eg 0.1 or -0.5, which is within one standard deviation of the mean of 0); you get -1 if you predict a number thats "far" from this distribution (something like 40 - which has an infinitesimally low probability of being drawn from a unit gaussian). mode collapse, then, is when you predict 0 all the time. yes, you are technically correct but youve failed to learn the true distribution. obviously ive simplified this quite a bit and have anthropomorphize the model, but i hope you get the gist. otherwise, the [original paper](https://arxiv.org/abs/1406.2661) is refreshingly easy to read.
This is likely part of how the speech-generating NNs are trained (ie. there's a generated-speech-detector and the network is trained to fool it, while it is also trained): https://arxiv.org/abs/1406.2661
So this is to counter stuff like GAN's (generative adversarial networks) [1] to reverse engineer data out of a black box systems? Like Yahoo's NSFW [2] classifier for example. [1] https://en.wikipedia.org/wiki/Generative_adversarial_network... [1] https://arxiv.org/abs/1406.2661 [2] https://github.com/yahoo/open_nsfw
My understanding is that the use of game theory in this context has nothing to do with the domain in which the network is being trained. It's not about creating game playing AI's—it's about a 'game' between two neural networks, one who competes by getting better at classifying, and the other by getting better at generating. That basic architecture is quite general—which I think is why you also see e.g. LeCun being excited about this direction. To be fair, the article does a kind of weird thing in discussing three different ways game theory has shown up in the context of machine learning recently—but they're all separate ideas really. Edit: a more precise description in the abstract here: https://arxiv.org/abs/1406.2661
your idea is called adversarial networks, and it works quite well (http://arxiv.org/abs/1406.2661) although it's not useful every time.
An overlapping group of authors (compared with the ones that wrote the "intriguing properties" paper) recently published a neat preprint on using adversarial networks as part of the training process. The goals of the paper are different from the networks discussed here, but I thought it was really interesting. http://arxiv.org/abs/1406.2661
> I'm disgusted, but not surprised, that teenage kids are generating CSAM like this. Even before we had diffusion models, we had GANs and deepfakes, which were almost immediately used for generating shittons of nonconsensual porn I think the big difference now is that 1) it's much easier to do now, and 2) the computational requirements and (more importantly) technical skills have dramatically dropped. We should also be explicitly aware that deep fakes are still new. GANs in 2014 were not creating high definition images. They were doing fuzzy black and white 28x28 faces, poorly, and 32x32 color images that if you squint hard enough you could see a dog…
(https://arxiv.org/abs/1406.2661). MNIST was a hard problem at that time and that's 10 years. It took another 4 years to get realistic faces and objects (https://arxiv.org/abs/1710.10196) (mind you, those images are not random samples), another year to get to high resolution, and another 2 to get to diffusion and another 2 before those exploded. Deep fakes were really only a thing within the last 5 years and certainly not on consumer hardware. I don't think the legal system moves much in 10 years let alone 5 or 2. I think a lot of us have not accurately encoded how quickly this whole space has changed. (image synthesis is my research area btw) I'm not surprised that these teenagers in a small town did this. But the fact that all those adjectives exist in that order is distinct. Discussions of deep fakes like that Tom Scott video were barely a warning (5 years is not a long time). It quickly went from researchers thinking it can happen in the next decade and starting discussions to real world examples making the news in under their prediction time (I don't think anyone expected how much money and man hours would be dumped into AI).
This article talks about technologies used to produce human-like text, and "deep-fake" video and audio. The article argues that such generative AI can be used by criminals to commit fraud- but at the same time, AI can also be used on the other side to detect and prevent fraud! You might be interested to know that the technological breakthrough that made today's realistic image and audio generation possible was in fact a form of this very idea: one AI creates fake but realistic data, while another AI attempts to detect the fake data. The two systems fight against each other in a game theoretical back-and-forth contest until both the fraudster and the fraud detector are…
almost perfect. The idea, the Generative Adversarial Network, was published by Goodfellow et al. [1] in 2014 and has spread widely since then. My favorite introductory explanation of GANs is [2], which describes the generator as a money counterfeiter attempting to fraudulently submit fake dollar bills, and the discriminator as a police officer who is inspecting each dollar bill to determine whether it is real or counterfeit. Given infinite time and memory, the mathematical result is always the same: the fraudster's output becomes perfect, and the police can only guess at random. [1] https://arxiv.org/abs/1406.2661 [2] https://medium.com/@ageitgey/abusing-generative-adversarial-...
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Scott Aaronson
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.