EGC: Image Generation and Classification via a Single Energy-Based Model

by   Qiushan Guo, et al.

Learning image classification and image generation using the same set of network parameters is a challenging problem. Recent advanced approaches perform well in one task often exhibit poor performance in the other. This work introduces an energy-based classifier and generator, namely EGC, which can achieve superior performance in both tasks using a single neural network. Unlike a conventional classifier that outputs a label given an image (i.e., a conditional distribution p(y|𝐱)), the forward pass in EGC is a classifier that outputs a joint distribution p(𝐱,y), enabling an image generator in its backward pass by marginalizing out the label y. This is done by estimating the energy and classification probability given a noisy image in the forward pass, while denoising it using the score function estimated in the backward pass. EGC achieves competitive generation results compared with state-of-the-art approaches on ImageNet-1k, CelebA-HQ and LSUN Church, while achieving superior classification accuracy and robustness against adversarial attacks on CIFAR-10. This work represents the first successful attempt to simultaneously excel in both tasks using a single set of network parameters. We believe that EGC bridges the gap between discriminative and generative learning.


page 1

page 7

page 8


Towards Bridging the Performance Gaps of Joint Energy-based Models

Can we train a hybrid discriminative-generative model within a single ne...

JGAN: A Joint Formulation of GAN for Synthesizing Images and Labels

Image generation with explicit condition or label generally works better...

MPG: A Multi-ingredient Pizza Image Generator with Conditional StyleGANs

Multilabel conditional image generation is a challenging problem in comp...

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One

We propose to reinterpret a standard discriminative classifier of p(y|x)...

MetalGAN: Multi-Domain Label-Less Image Synthesis Using cGANs and Meta-Learning

Image synthesis is currently one of the most addressed image processing ...

Energy-Inspired Self-Supervised Pretraining for Vision Models

Motivated by the fact that forward and backward passes of a deep network...

ReluDiff: Differential Verification of Deep Neural Networks

As deep neural networks are increasingly being deployed in practice, the...

Please sign up or login with your details

Forgot password? Click here to reset