2018

Skill Rating for Generative Models

Olsson, Catherine, Bhupatiraju, Surya, Brown, Tom et al.

Understand

We explore a new way to evaluate generative models using insights from evaluation of competitive games between human players.

We show experimentally that tournaments between generators and discriminators provide an effective way to evaluate generative models.
We introduce two methods for summarizing tournament outcomes: tournament win rate and skill rating.
Evaluations are useful in different contexts, including monitoring the progress of a single model as it learns during the training process, and comparing the capabilities of two different fully trained models.

Built on

The rating of chessplayers, past and present
A. Elo · 1978
Earlier work this paper cites.
A comprehensive guide to chess ratings
M. Glickman · 1995
Earlier work this paper cites.
Trueskill: a bayesian skill rating system
Ralf Herbrich, Tom Minka, and Thore Graepel · 2007
Earlier work this paper cites.
Unlearning for better mixing
Olivier Breuleux, Yoshua Bengio, and Pascal Vincent · 2010
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
M. Gutmann and A. Hyvarinen · 2010
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng · 2011
Earlier work this paper cites.
Example of the glicko-2 system
M. Glickman · 2013
Earlier work this paper cites.
On distinguishability criteria for estimating generative models
Ian J. Goodfellow · 2014
Earlier work this paper cites.
Generative adversarial networks
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Deep generative image models using a Laplacian pyramid of adversarial networks
Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus · 2015
Earlier work this paper cites.

Similar

Unsupervised representation learning with deep convolutional generative adversarial networks
Original
A. Radford, L. Metz, and S. Chintala · 2015
Cited alongside, same era.
Stop citing ross et al. 2010, “who are the crowdworkers”?
S. Silberman, K. Milland, R. LaPlante, J. Ross, and L. Irani · 2015
Cited alongside, same era.
Rethinking the Inception Architecture for Computer Vision
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna · 2015
Cited alongside, same era.
Generating images with recurrent adversarial networks
Original
Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic · 2016
Cited alongside, same era.
Improved techniques for training gans
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Cited alongside, same era.
A note on the evaluation of generative models
Original
L. Theis, A. van den Oord, and M. Bethge · 2016
Cited alongside, same era.
Generative multi-adversarial networks
Original
Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan · 2017
Cited alongside, same era.
An online learning approach to generative adversarial networks
Original
P. Grnarova, K. Levy, A. Lucchi, T. Hofmann, and A. Krause · 2017
Cited alongside, same era.

Then

Improved training of wasserstein gans
I. Gulrajani · 2017
Later among the works it cites.
A neural representation of sketch drawings
Original
D. Ha and D. Eck · 2017
Later among the works it cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter · 2017
Later among the works it cites.
Progressive growing of gans for improved quality, stability, and variation
Original
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen · 2017
Later among the works it cites.
More on dota 2
OpenAI · 2017
Later among the works it cites.
Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications
Original
T. Salimans, A. Karpathy, X. Chen, and D. Kingma · 2017
Later among the works it cites.
Do GANs learn the distribution? some theory and empirics
Sanjeev Arora, Andrej Risteski, and Yi Zhang · 2018
Closest in time.
Pros and cons of gan evaluation measure
Original
A. Borji · 2018
Closest in time.
A classification-based perspective on GAN distributions, 2018
Shibani Santurkar, Ludwig Schmidt, and Aleksander Madry · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…