Automatic Differentiation Variational Inference

I truly believe that Bayesian inference is the statistics of the 21st century. Recent advances in MCMC (e.g., NUTS, Stan [1]) and variational inference (e.g., ADVI [2], VAE [3], etc.) + more computing power than ever promise a near future in which Bayesian inference is the default inference engine. Prior distribution is a beautiful and logical mechanism for adding regularization, domain-specific knowledge to our model. [1] Stan, a platform for statistical modeling http://mc-stan.org/ [2] Automatic Differentiation Variational Inference https://arxiv.org/abs/1603.00788 [3] Auto-Encoding Variational Bayes https://arxiv.org/abs/1312.6114

No, back-propagation would not give full Bayesian inference (although there are some tricks [0]). They instead use variational inference[1], which allows for fast inference of continuous PGMs. [0] http://mlg.eng.cam.ac.uk/yarin/blog_2248.html [1] https://arxiv.org/abs/1603.00788

There are two lines of research in this direction. So there is work from my lab [1] and some folks at ETH-Zurich [2] is automatically finding closed form solutions. These when used do give performance equivalent to handwritten methods. Beyond that we are seeing some cool stuff with Blackbox Variational Inference [3] and other Automatic Variational [4] solvers coming out of Blei's lab. At present these capabilities are spread about the different system but I expect all them to eventually end up available in whichever you choose to use. [1] http://homes.soic.indiana.edu/ccshan/rational/simplify-padl.... [2] http://www.srl.inf.ethz.ch/papers/psi-solver.pdf [3] http://www.cs.columbia.edu/~blei/papers/RanganathGerrishBlei... [4] http://arxiv.org/pdf/1603.00788v1.pdf