The Bayesian central limit theorem for exponential family distributions: a geometric approach

The Bernstein-von Mises theorem, also known as the Bayesian Central Limit Theorem (BCLT), states that under certain assumptions a posterior distribution can be approximated as a multivariate normal distribution as long as the precision parameter is large. We derive a special case of the BCLT for the canonical conjugate prior of a regular exponential family distribution using the machinery of information geometry. Our approach applies the core approximation for the BCLT, Laplace’s method, to the free entropy (i.e., log-normalizer) of an exponential family distribution. Additionally, we formulate approximations for the Kullback–Leibler divergence and Fisher-Rao metric on the conjugate prior manifold in terms of corresponding quantities from the likelihood manifold. We also include an application to the categorical distribution and show that the free entropy derived approximations are related to various series expansions of the gamma function and its derivatives. Furthermore, for the categorical distribution, the free entropy approximation produces higher order expansions than the BCLT alone.