Variational Inference: A Review for Statisticians

To grok the ELBO, don't start with VAEs; start with basic variational inference. https://arxiv.org/abs/1601.00670 is good, though maybe increasingly dated...Once you're happy with VI in general, https://arxiv.org/abs/1606.05908 is a nice read on the VAE.

> I might take up your challenge and write a blog post about it or something if I get any success. Success or failure, I would really enjoy seeing that write up! I would be even more excited to be proven wrong. The promise of "universal function approximator" is very temping. My personal dream would be to have it so one could essentially run scipy in reverse and learn the entire library with a NN. Even in the case I gave, the idea that you could learn an arbitrary quantile function means you could also arbitrarily learn a sampler for any distribution, since all you have to do compose a uniform sampler with whatever quantile function you learned. Of course for this example I'm using "solved" using a similar approach with variational inference (pyro has a write up on it: https://pyro.ai/examples/svi_part_i.html, you might find David Blei's "Variational Inference: A Review for Statisticians" useful as well https://arxiv.org/abs/1601.00670)

Probabilistic programming uses computer science techniques to do automated statistical modeling. For example, imagine I have a coin, and I want to discover if it is biased, i.e. if it lands on heads more often than tails. In a probabilistic programming framework, I can express my model as a simple Bernoulli model, `x ~ Bernoulli(p)`, and then automatically estimate the bias parameter `p` given some data (do "inference"). You can easily do this calculation by hand or in Python, but this does not generalize to more complex real-world scenarios. For complex probabilistic models, we must rely on numerical approximations. MCMC is just one algorithm for doing this approximate inference. Another popular technique is called variational inference [2]. Another commenter mentioned HMC [3], which is just a specific instance of MCMC. [1] https://mc-stan.org/ [2] https://arxiv.org/abs/1601.00670 [3] https://arxiv.org/abs/1206.1901

While I am not statistically sophisticated to say whether the statement is true or not, it seems that the mathematical machinery is strongly converging, as witnessed by papers like [1], where some of the statistical machinery is being developed on the ML side. It might be historically interesting to note that 20-30 years ago, ML also more evenly spanned work that would be considered AI based -- as including logic or knowledge representation see [2 - 4] as examples. [1] Variational Inference: A Review for Statisticians, https://arxiv.org/abs/1601.00670 [2] A Summary of Machine Learning Papers from IJCAI-85 http://web.engr.oregonstate.edu/~tgd/publications/mlj-ijcai8... [3] Chunking in soar: the anatomy of a general learning mechanism, http://repository.cmu.edu/cgi/viewcontent.cgi?article=2552&c... [4] Explanation-Based Learning: An Alternative View, http://www.cs.utexas.edu/~ml/papers/ebl-mlj-86.pdf