This post begins with an apparent contradiction: on the one hand, the
reparameterization trick seems limited to a handful of distributions;
on the other, every random variable we simulate on our computers is
ultimately a reparameterization of a bunch of uniforms. So what
gives? Our investigation into this question led to the paper,
“Reparameterization Gradients through Acceptance-Rejection Sampling
Algorithms,” which we recently presented at AISTATS . In it, we
debunk the myth that the gamma distribution and all the distributions
that are derived from it (Dirichlet, beta, Student’s t, etc.) are not
amenable to reparameterization [2-5]. We’ll show how these
distributions can be incorporated into automatic variational inference
algorithms with just a few lines of Python.
A few weeks ago we read and discussed two papers extending the Variational Autoencoder (VAE) framework: “Importance Weighted Autoencoders” (Burda et al. 2016) and “Adversarial Autoencoders” (Makhzani et al. 2016). The former proposes a tighter lower bound on the marginal log-likelihood than the variational lower bound optimized by standard variational autoencoders. The latter replaces the KL divergence term — between the approximate posterior and prior distributions over latent codes — in the variational lower bound with a generative adversarial network (GAN) that encourages the aggregated posterior to match the prior distribution. In doing so, both extensions aim to improve the standard VAE’s ability to model in complex posterior distributions.
In this week’s session, Yixin led our discussion of two papers about Generative Adversarial Networks (GANs). The first paper,
“Generalization and Equilibrium in Generative Adversarial Nets” by Arora et al. , is a theoretical investigation
of GANs, and the second paper, “Improved Training of Wasserstein GANs” by Gulrajani et al. , gives an new training
method of Wasserstein GAN. This video gives a good explanation of the first paper.