Learning Unnormalized Statistical Models via Compositional Optimization

by   Wei Jiang, et al.

Learning unnormalized statistical models (e.g., energy-based models) is computationally challenging due to the complexity of handling the partition function. To eschew this complexity, noise-contrastive estimation (NCE) has been proposed by formulating the objective as the logistic loss of the real data and the artificial noise. However, as found in previous works, NCE may perform poorly in many tasks due to its flat loss landscape and slow convergence. In this paper, we study it a direct approach for optimizing the negative log-likelihood of unnormalized models from the perspective of compositional optimization. To tackle the partition function, a noise distribution is introduced such that the log partition function can be written as a compositional function whose inner function can be estimated with stochastic samples. Hence, the objective can be optimized by stochastic compositional optimization algorithms. Despite being a simple method, we demonstrate that it is more favorable than NCE by (1) establishing a fast convergence rate and quantifying its dependence on the noise distribution through the variance of stochastic estimators; (2) developing better results for one-dimensional Gaussian mean estimation by showing our objective has a much favorable loss landscape and hence our method enjoys faster convergence; (3) demonstrating better performance on multiple applications, including density estimation, out-of-distribution detection, and real image generation.


page 8

page 9

page 14

page 15


Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation

Noise-contrastive estimation (NCE) is a statistically consistent method ...

Fast and Simple Optimization for Poisson Likelihood Models

Poisson likelihood models have been prevalently used in imaging, social ...

A Family of Computationally Efficient and Simple Estimators for Unnormalized Statistical Models

We introduce a new family of estimators for unnormalized statistical mod...

Conditional Noise-Contrastive Estimation of Unnormalised Models

Many parametric statistical models are not properly normalised and only ...

Pitfalls of Gaussians as a noise distribution in NCE

Noise Contrastive Estimation (NCE) is a popular approach for learning pr...

Score Matching Model for Unbounded Data Score

Recent advance in score-based models incorporates the stochastic differe...

Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency

Noise Contrastive Estimation (NCE) is a powerful parameter estimation me...

Please sign up or login with your details

Forgot password? Click here to reset