Truncated Variational Expectation Maximization
share
We derive a novel variational expectation maximization approach based on truncated variational distributions. The truncated distributions are proportional to exact posteriors in a subset of a discrete state space and equal zero otherwise. In contrast to factored variational approximations or Gaussian approximations, truncated approximations neither assume posterior independence nor mono-modal posteriors. The novel variational approach is closely related to Expectation Truncation (Lücke and Eggert, 2010) - a preselection based EM approximation. It shares with Expectation Truncation the central idea of truncated distributions and the application domain of discrete hidden variables. In contrast to Expectation Truncation we here show how truncated distributions can be included into the theoretical framework of variational EM approximations. A fully variational treatment of truncated distributions then allows for derivations of novel general and mathematically grounded results, which in turn can be used to formulate novel efficient algorithms for parameter optimization of probabilistic data models. Apart from showing that truncated distributions are fully consistent with the variational free-energy framework, we find the free-energy that corresponds to truncated distributions to be given by compact and efficiently computable expressions, while update equations for model parameters (M-steps) remain in their standard form. Furthermore, expectation values w.r.t. truncated distributions are given in a generic form. Based on these observations, we show how an efficient and easily applicable meta-algorithm can be formulated that guarantees a monotonic increase of the free-energy. More generally, the obtained variational framework developed here links variational E-steps to discrete optimization, and it provides a theoretical basis to tightly couple sampling and variational approaches.
READ FULL TEXT