Local Max-Entropy and Free Energy Principles Solved by Belief Propagation
share
A statistical system is classically defined on a set of microstates E by a global energy function H : E →ℝ, yielding Gibbs probability measures (softmins) ρ^β(H) for every inverse temperature β = T^-1. Gibbs states are simultaneously characterized by free energy principles and the max-entropy principle, with dual constraints on inverse temperature β and mean energy U(β) = 𝔼_ρ^β[H] respectively. The Legendre transform relates these diverse variational principles which are unfortunately not tractable in high dimension. The global energy is generally given as a sum H(x) = ∑_ a ⊂Ω h_ a(x_| a) of local short-range interactions h_ a : E_ a→ℝ indexed by bounded subregions a⊂Ω, and this local structure can be used to design good approximation schemes on thermodynamic functionals. We show that the generalized belief propagation (GBP) algorithm solves a collection of local variational principles, by converging to critical points of Bethe-Kikuchi approximations of the free energy F(β), the Shannon entropy S( U), and the variational free energy F(β) = U - β^-1 S( U), extending an initial correspondence by Yedidia et al. This local form of Legendre duality yields a possible degenerate relationship between mean energy U and β.
READ FULL TEXT