Accelerating Natural Gradient with Higher-Order Invariance

An overview for our ICML 2018 paper, Accelerating Natural Gradient with Higher-Order Invariance. Natural gradient update loses its invariance due to the finite step size. In this paper, we study the invariance of natural gradient from the perspective of Riemannian geometry, and propose several new update rules to improve its invariance. Empirical results show that better invariance can result in faster convergence in several supervised learning, unsupervised learning and reinforcement learning applications.

A-NICE-MC: Adversarial Training for MCMC

A brief introduction to our NIPS paper, A-NICE-MC: Adversarial Training for MCMC. In this paper, we introduce a method that allows us to train domain specific MCMC proposals that are more efficient than hand crafted ones.

Variational Rejection Sampling

An overview for our upcoming AISTATS 2018 paper, Variational Rejection Sampling. Variational learning of intractable posteriors is computationally efficient, but the resulting approximations can be poor. We propose a general approach to increase the flexibility of any variational family by augmenting a parameterized variational posterior with a differentiable and tunable accept-reject step. The rejection rate for the resulting resampled posterior can be adaptively controlled to monotonically trade-off computation for statistical accuracy, with the resampled posterior matching the true posterior in the limit.

Approximate Inference via Weighted Rademacher Complexity

Highlights from our AAAI 2018 paper, "Approximate Inference via Weighted Rademacher Complexity." In this work we consider the challenging problem of computing the sum of more numbers than can be explicitly enumerated. This sum arises in various contexts, such as the partition function of a graphical model, the permanent of a matrix, or the number of satisfying assignments of a propositional formula. By establishing a novel connection with Rademacher complexity, we show how this sum can be estimated and bounded by solving an optimization problem; finding the largest number in the sum after random perturbations have been applied.

Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models

An overview for our upcoming AAAI 2018 paper, Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models. We propose a new approach to evaluate, compare, and interpolate between objectives based on maximum likelihood estimation and advesarial training for learning generative models. We find that even though adversarial training generates visually appealing samples, it obtains log-likelihoods that are orders of magnitudes worse than maximum likelihood -- even a trivial Gaussian mixture model baseline memorizing the data can obtain better likelihoods (and beautiful samples)! But, why?

A Tutorial on Information Maximizing Variational Autoencoders (InfoVAE)

This tutorial discusses MMD variational autoencoders, a member of the InfoVAE family. It is an alternative to traditional variational autoencoders that is fast to train, stable, and easy to implement.
Shengjia Zhao

Learning Hierarchical Features from Generative Models

Current ways of stacking variational autoencoders may not always provide meaningful structured features. In fact, we showed in a recent ICML paper that while existing approaches have shortcomings, a new ladder architecture can often learn distentangled features.