This week we discussed MADE (Germain et al., 2015 [1]) and NADE (Uria et al., 2016 [2]), two papers on autoregressive distribution estimation. The two papers take a similar approach to estimating the distributions of data. Namely, they modify the structure of autoencoder neural networks to yield properly normalized, autoregressive models. NADE, first introduced in 2011, lays the groundwork for these models. MADE extends these ideas to deep networks with binary data. The recent, journal paper on NADE further extends these ideas to real valued data, explores more interesting network architectures, and performs more extensive experiments.

This week, we read Gal and Ghahramani’s “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning” [1], as well as “Deep Networks with Stochastic Depth” by Huang et. al [2]. The two papers differed greatly in scope: Gal and Ghahramani looked at Dropout from a Bayesian point of view and cast it as approximate Bayesian inference in deep Gaussian Processes while Huang et al. demonstrated the possibility of incorporating Dropout in the Resnet architecture.

On our reading list for this week we had the paper by Advani and Ganguli titled “Statistical Mechanics of Optimal Convex Inference in High Dimensions” [1]. Classical statistics gives answers to questions on the accuracy with which statistical models can be inferred in the limits where the measurement density is $\alpha=\frac{N}{P}\rightarrow\infty$. In this work the authors are concerned with the analyses of the fundamental limits in the finite $\alpha$ regime which is the “big data” regime where both the number of data points $N$ and the number of parameters $P$ tend to infinity but their ratio is still finite.