Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

In their tech report, they say this is based on: > "Our methods extend [28] through careful modifications to the data and computation to scale up learning." [28] is Lou et al. (2023), the "Score Entropy Discrete Diffusion" (SEDD) model (https://arxiv.org/abs/2310.16834). I wrote the first (as far as I can tell) independent from-scratch reimplementation of SEDD: https://github.com/mstarodub/dllm My goal was making it as clean and readable as possible. I also implemented the more complex denoising strategy they described (but didn't implement). It runs on a single GPU in a few hours on a toy dataset.

Thanks, yes, I was thinking specifically of "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution". They actually consider two noise distributions: one with uniform sampling for each noised token position, and one with a terminal masking (the Q^{uniform} and Q^{absorb}.) However, the terminal-masking system is clearly superior in their benchmarks. https://arxiv.org/pdf/2310.16834#page=6

Here are two papers linked from Inception's site: 1. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution - https://arxiv.org/abs/2310.16834 2. Simple and Effective Masked Diffusion Language Models - https://arxiv.org/abs/2406.07524

Few details about the model have been made public, as far as I know. However, Steano Ermon, one of the founders of Inception Labs, the company that released the model, co-authored a paper in late 2023 (revised in 2024), titled "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution." The authors tested their proposed method on small autoregressive language modeling tasks, and claimed promising results: Paper: https://arxiv.org/abs/2310.16834 Code: https://github.com/louaaron/Score-Entropy-Discrete-Diffusion Blog post: https://aaronlou.com/blog/2024/discrete-diffusion/ It's possible -- in fact, likely -- that the new model is a much larger, more refined version of the small model presented and tested in that paper.

Not today, but we will be following up with a technical report over the next week or so. In the meantime, you can take a look at some of the research papers that inspired our work: - https://arxiv.org/abs/2310.16834 - https://arxiv.org/abs/2406.07524

Thanks for bringing this up! As far as I understand it current text diffusion models are limited to fairly short context windows. The idea of a text diffusion model continuously updating and revising a million-token-long chain-of-thought is pretty mind-boggling. I agree that these non-autoregressive models could potentially behave in completely different ways. That said, I'm pretty sure we're a long way from building equally-competent diffusion-based base models, let alone reasoning models. If anyone's interested in this topic, here are some more foundational papers to take a look at: - Simple and Effective Masked Diffusion Language Models [2024] (https://arxiv.org/abs/2406.07524) - Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [2023] (https://arxiv.org/abs/2310.16834) - Diffusion-LM Improves Controllable Text Generation [2022] (https://arxiv.org/abs/2205.14217)

The SEDD architecture [1] probably allows for parallel sampling of all tokens in a block at once, which may be faster but not necessarily less computationally demanding in terms of runtime times computational resources used. [1] Which Inception Labs's new models may be based on; one of the cofounders is a co-author. See equations 18-20 in https://arxiv.org/abs/2310.16834