Auto-Encoding Variational Bayes

I truly believe that Bayesian inference is the statistics of the 21st century. Recent advances in MCMC (e.g., NUTS, Stan [1]) and variational inference (e.g., ADVI [2], VAE [3], etc.) + more computing power than ever promise a near future in which Bayesian inference is the default inference engine. Prior distribution is a beautiful and logical mechanism for adding regularization, domain-specific knowledge to our model. [1] Stan, a platform for statistical modeling http://mc-stan.org/ [2] Automatic Differentiation Variational Inference https://arxiv.org/abs/1603.00788 [3] Auto-Encoding Variational Bayes https://arxiv.org/abs/1312.6114

This is the PhD thesis of Durk Kingma [1]. Reading this right now; appears to be a well-written introduction to stuff variational autoencoders [2] and ADAM [3]. [1] http://dpkingma.com/ [2] http://arxiv.org/abs/1312.6114 [3] http://arxiv.org/abs/1412.6980

Causality is a bit harder to integrate with current machine learning models as it's hard even with standard probabilistic graphical models. On the other hand, there has been a lot of work integrating deep neural networks with probabilistic models. For example, the variational auto-encoders are a graphical model with Gaussian latent variables whose mean and variance are determined by (deep) neural networks [1]. There has been work exploring the neural network weights as latent variables themselves [2]. Finally, some new developments such as dropout can be interpreted as some form of deep Gaussian processes [3]. I believe there will be a lot further developments on this area in the near-future. [1] http://arxiv.org/abs/1312.6114 [2] http://arxiv.org/abs/1505.05424 [3] http://arxiv.org/abs/1506.02142