We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x,y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures may beused and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, andout-of-distribution detection while also enabling our models to generate samplesrivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and presentan approach which adds little overhead compared to standard classification training. Our approach is the first to achieve performance rivaling the state-of-the-artin both generative and discriminative learning within one hybrid model.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Thanks. I am passionate haha. I do like to educate, but it is often difficult to do on these forms as I do not know peoples backgrounds. If you throw too high level at them, I find it does more harm than good. I also find that while I know many here took things like calculus, that we also need to recognize that skills degrade with time (though can be regained more easily).
I do understand that it is difficult to find the math pathways in ML. What I do suggest is focusing on generative models[0], but the important part is to keep asking "why" and "what does this actually mean?" I think it is too easy to get stuck in knowing the answer and accepting it without understanding…
Read the full comment
that there is no "right" answer, but less wrong answers. The path gets clearer if this attitude is adopted. This is where I've found teaching highly beneficial[1], as you face the basics and you'll find many questions that are easy to dismiss. To be a good teacher you have to take dumb questions seriously to determine if they're actually dumb or "dumb"[2]. Things you probably asked when learning: "how many layers?", "how many neurons per layer?", "is the optimization space actually smooth like the 2d decent graphs we saw?", "does data lie on a manifold?", "what is a manifold", "does data always lie on a lower dimensional manifold?", "is data always a distribution?", "is my data actually representative of my goals?", "what does this measurement mean?", "what does this measurement not tell me?", and so on. These are all extremely important questions that are almost universally ignored, but that all appear to have simple answers.
The reason I do like category theory (Bartosz also has video lectures for those), is because it helps connect the many different disciplines of math that are needed to answer some of these. To see the generalizations of things like surjective (epimorphic) and injective (monomorphic) functions, plays a role in answering the layer and neuron questions. It was the way I could start understanding how field theory wasn't just cool but impractical math.
But to ML, I think there's this hard gap and I'm not sure of a good resource that fills it (I'm working on one myself). That there's lots of basics (blogs like "the math behind transformers" that show the equations and little to no more) and there's also plenty at a high level by experts in cat theory, set theory, algebraic geometry, or others. The former aren't very useful and the latter can be easily found if you have the requisite knowledge but are impenetrable if you don't.
But with diffusion models and score matching being all the rage now, I highly suggest reading into Aapo Hyvärinen's[3] work. At lower levels I suggest Gelman's book and/or McElreath's. Needham has impressive illustrations and writes so well that you can only emulate because he's at a different level. I found Shao's Mathematical Statistics greatly helpful, but this is not easy to parse. Gallier and Quaintance are also worth looking into. But if you need something on the easier side, Tomczak is your friend. On the YouTube side I'll recommend some that are more easily missed: mathemaniac, EpsilonDeltaMain, jHan, ron-math, and Mutual_Information. You should find more from these too. Also search the #some{,2,3} tags, there's a lot of hidden gems. There's also a Cats4AI group btw, and many of them have now created a startup.
[0] Realistically all models are generative. The definition is ill-defined and you can even demonstrate that classifiers are EBMs. https://arxiv.org/abs/1912.03263
[1] I'm finishing a PhD program, so I do teach a ML course
[2] In either case you have to answer nicely. But it is easy to trick yourself into thinking something is simple when it is not.
[3] https://www.cs.helsinki.fi/u/ahyvarin/papers/
But BERT isn't a generative model? Unless you mean it's trained similar to a generative model by predicting the missing tokens?
Also an interesting paper about hybrid energy based models:
https://arxiv.org/pdf/1912.03263.pdf
It's possible to have a model be both a classifier + generative, and the performance is comparable (although lower) to the SOTA of single-purpose models.
For people interested in the softmax, log sum exp and energy models, have a look at "Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One" [1]
[1]: https://arxiv.org/abs/1912.03263
For neural networks unless it's calibrated otherwise the confidence score has little meaning to the accuracy. [1] see figure 4 and you will understand what I mean
[1] https://arxiv.org/pdf/1912.03263.pdf
Reminds me of JEMs from Grathwohl et al last year [0]. They train a generative model by treating it an an Energy Based Model using stochastic langevin gradients. I’m curious as to how it relates to the models in this post.
[0] https://arxiv.org/abs/1912.03263
> The rule in ML There's definitely attempts to revive things (in the general sense, not just GANs), but most successes appear to come from large labs dedicating equal computational resources to the older models and often by changing names. This can make things more confusing and make things appear to be changing faster, but once you can see this, you'll have an easier time keeping up (so being new, watch out for this). I'll give some examples that are easier to read[0-2] (i.e. don't need expert knowledge to understand the nuances).
As an insider (ML researcher), my complaint isn't so much about that we have a large proportion of people chasing one specific avenue, it is…
Read the full comment
that we gate keep newer methods. I think you can see a similar effect on HN when new models are proposed. They are trashed due to lack of beating existing models (this is true even beyond ML!). There will always be reasons to critique works and I don't want discourage criticism, but I do want to discourage dismissal. It hinders progress, because progress is made in small steps, not leaps and bounds. I think this can get confusing for someone entering the field (I'm sorry if I've misjudged, I'm inferring from the comments).
> On the FFHQ point This is an excellent question that unfortunately I don't know the answer to. I think you'll find this work helpful[3], it has the largest human study. But despite its name, StyleNAT performed best in specifically FFHQ. What I would say is that there are good arguments to make the diffusion models are better at representing a diversity of images (making them well suited for things like art generation) but theoretically GANs are approximating your full density distribution. There's some talks by Goodfellow discussing this but I don't recall which ones.
> I didnt know they wrote custom kernels As you've probably found, the StyleGAN code is not the easiest to read lol. Since you're using pytorch, you can find them here[4]. I encourage you to look at these, especially if you've never seen CUDA code before. Because the biggest takeaway will be that you'll see how easy it is to add a custom kernel, and given the earlier comment you'll see the utility ^__^
I'm not going to discourage you from trying triton, but I'll note that pytorch's compile goes a long way. Definitely _start_ there (see TensorRT).
> What does it mean to have a backbone? Does it just mean the underlying architecture used in the method? Exactly! So in the case of a diffusion model it is the UNet (the neural network part, and specifically this part estimates the parameters for the probability distribution. If this doesn't make sense now, it will later. If you are struggling to understand diffusion models after spending some time reading the papers, come back to this comment). You'll also find the term "backbone" used in application based models such as in Semantic Segmentation, Object Detection, Pose Estimation, and much more. In those cases, these are typically pretrained, so recognize this as a hyper-parameter.
> Also, on the decoder only vs encoder-decoder point I'm going to say something frustrating. In short: yes. If we get a but more nuanced: no. If we get really nuanced: yes. I know this isn't a great answer, but it can be really difficult to understand. On the surface, yes because you need to encode the variable and your model needs to transform a dimension starting at R^N and ending in R^N. While a (I have to stress, colloquial[5]) GAN transforms a R^M space to R^N where M << N. With more nuance you can argue it is the backbone. But to be detailed, you'll find that there are fundamental factors placing computational bounds on the theoretical performance of these architectures. To get there you'll need to carefully study Goodfellow's original paper (some follow-ups expanding on the analysis) as well as the "original" diffusion paper by Sohl-Dickstein[6] (quotes because this is debatable, but the claim has reasonable merit), and you should become familiar with Aapo Hyvärinen[7]. The last is by far the hardest part and the confusion is normal. I know quite a few well renowned and intelligent people who struggled (personally I went through the "this is hard", "this is easy", "ah fuck, I actually don't understand anything" cycle for a bit. But that's just a signal that you're progressing :).
> Thanks for the detailed comment, you've given me a lot to think about. Great! I hope this can help provide direction (I read your other post). The best and worth thing about machine learning is that there's so much depth. It can both be intimidating and easy to miss. But if you're passionate about learning (as it appears) you'll find the knowledge gained is highly rewarding, if unfortunately hard to gain.
And I apologize for being so verbose. It is a bad habit.
[0] Diffusion Models Beat GANs on Image Synthesis (https://arxiv.org/abs/2105.05233) The two authors are rockstars. Given your blog post, I think you'll enjoy a lot of their work. Which includes diffusion.
[1] ResNet strikes back: An improved training procedure in timm (https://arxiv.org/abs/2110.00476) Again, all three are great people to follow. You won't see many papers by Wightman, but you'll see his work with (now) Hugging Face. Notably he's one of the most important players in ViTs.
[2] A ConvNet for the 2020s (https://arxiv.org/abs/2201.03545) and ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders (https://arxiv.org/abs/2301.00808)
[3] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models (https://arxiv.org/abs/2306.04675)
[4] https://github.com/NVlabs/stylegan2-ada-pytorch/tree/main/to...
[5] I'm sorry, I still have difficulties explaining this, especially simply. There's a lot of points here. But one is easy to understand and what I mentioned before: GAN is a training method, not an architecture. A bit more nuance will be found by reading this far underappreciated work: https://arxiv.org/abs/1912.03263. The last point I want to mention is to never forget that "generative" is a general term and these models are good for generating __data__. Images are data, but to think this is the only type of data a GAN (or any model. I literally mean any[3]) can generate is naive. All of this gets harder to explain and I don't have the skill to do so in a simple manner and am afraid it will just come off as a rant.
[6] Deep Unsupervised Learning using Nonequilibrium Thermodynamics (https://arxiv.org/abs/1503.03585)
[7] "Score matching" and "Noise-contrastive estimation" will be the most beneficial https://www.cs.helsinki.fi/u/ahyvarin/papers/
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.