Skill Rating for Generative Models

by   Catherine Olsson, et al.

We explore a new way to evaluate generative models using insights from evaluation of competitive games between human players. We show experimentally that tournaments between generators and discriminators provide an effective way to evaluate generative models. We introduce two methods for summarizing tournament outcomes: tournament win rate and skill rating. Evaluations are useful in different contexts, including monitoring the progress of a single model as it learns during the training process, and comparing the capabilities of two different fully trained models. We show that a tournament consisting of a single model playing against past and future versions of itself produces a useful measure of training progress. A tournament containing multiple separate models (using different seeds, hyperparameters, and architectures) provides a useful relative comparison between different trained GANs. Tournament-based rating methods are conceptually distinct from numerous previous categories of approaches to evaluation of generative models, and have complementary advantages and disadvantages.


Using Skill Rating as Fitness on the Evolution of GANs

Generative Adversarial Networks (GANs) are an adversarial model that ach...

Learning Robust Representations Of Generative Models Using Set-Based Artificial Fingerprints

With recent progress in deep generative models, the problem of identifyi...

The Evaluation of Rating Systems in Online Free-for-All Games

Online competitive games have become increasingly popular. To ensure an ...

Application-Oriented Benchmarking of Quantum Generative Learning Using QUARK

Benchmarking of quantum machine learning (QML) algorithms is challenging...

Graph Embedding Augmented Skill Rating System

This paper presents a framework for learning player embeddings in compet...

Demystifying Randomly Initialized Networks for Evaluating Generative Models

Evaluation of generative models is mostly based on the comparison betwee...

A State-Space Perspective on Modelling and Inference for Online Skill Rating

This paper offers a comprehensive review of the main methodologies used ...

Please sign up or login with your details

Forgot password? Click here to reset