Locally Typical Sampling

Too much money being thrown around on BS in the LLM space, hardly any of it is going to places where it matters. Ignorance on the part of investors. For example, the researchers working hard on better text sampling techniques (i.e. https://arxiv.org/abs/2202.00666), or on better constraint techniques (i.e. like this https://arxiv.org/abs/2306.03081), or on actual negative prompting/CFG in LLMs (i.e. like this https://github.com/huggingface/transformers/issues/24536) are doing far FAR more to advance the state of AI than dozens of VC backed LLM companies operating today. They are all laboring in relative obscurity. HN, and the NLP community have some serious blindspots with knowing how to exploit their own technology. At least someone at Andreessen Horowitz got a clue and gave some funding to Oogabooga - still waiting for Automatic1111 to get any funding.

> That's called beam search. There are other decoding approaches, other than beam search: https://arxiv.org/abs/2202.00666 > Other than that, things like gpt can't go back and edit what they've outputted Actually, they can, they may emit a backspace token [1] [1] https://arxiv.org/abs/2306.05426, https://news.ycombinator.com/item?id=36425375

Things like that probably require some kind of thinking ahead, which models of things kind kind of can't do-- something like beam search. Language models that utilise beam search can calculate integrals ('Deep learning for symbolic mathematics', Lample, Charton, 2019, https://openreview.net/forum?id=S1eZYeHFDS), but without it it doesn't work. However, beam search makes bad language models. I got linked this paper ('Locally typical sampling' https://arxiv.org/pdf/2202.00666.pdf) when I asked some people why beam search only works for the kind of stuff above. I haven't fully digested it though.