Temporally Coherent GANs for Video Super-Resolution (TecoGAN)

by   Mengyu Chu, et al.
Technische Universität München

Adversarial training has been highly successful in the context of image super-resolution. It was demonstrated to yield realistic and highly detailed results. Despite this success, many state-of-the-art methods for video super-resolution still favor simpler norms such as L_2 over adversarial loss functions. This is caused by the fact that the averaging nature of direct vector norms as loss functions leads to temporal smoothness. The lack of spatial detail means temporal coherence is easily established. In our work, we instead propose an adversarial training for video super-resolution that leads to temporally coherent solutions without sacrificing spatial detail. In our generator, we use a recurrent, residual framework that naturally encourages temporal consistency. For adversarial training, we propose a novel spatio-temporal discriminator in combination with motion compensation to guarantee photo-realistic and temporally coherent details in the results. We additionally identify a class of temporal artifacts in these recurrent networks, and propose a novel Ping-Pong loss to remove them. Quantifying the temporal coherence for image super-resolution tasks has also not been addressed previously. We propose a first set of metrics to evaluate the accuracy as well as the perceptual quality of the temporal evolution, and we demonstrate that our method outperforms previous work by yielding realistic and detailed images with natural temporal changes.


page 1

page 6

page 7

page 8

page 10

page 11

page 16

page 17


Photorealistic Video Super Resolution

With the advent of perceptual loss functions, new possibilities in super...

iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks

Recently, learning-based models have enhanced the performance of single-...

tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow

We propose a temporally coherent generative model addressing the super-r...

Perceptual Learned Video Compression with Recurrent Conditional GAN

This paper proposes a Perceptual Learned Video Compression (PLVC) approa...

Robust Super-Resolution GAN, with Manifold-based and Perception Loss

Super-resolution using deep neural networks typically relies on highly c...

A Multi-Pass GAN for Fluid Flow Super-Resolution

We propose a novel method to up-sample volumetric functions with generat...

HRVGAN: High Resolution Video Generation using Spatio-Temporal GAN

In this paper, we present a novel network for high resolution video gene...

Code Repositories


This repo contains source code and materials for the TEmporally COherent GAN SIGGRAPH project.

view repo


This is a fork of the TecoGAN project (https://github.com/thunil/TecoGAN) that adds support for docker.

view repo


Pytorch implementation of the TecoGan video super resolution model.

view repo

Please sign up or login with your details

Forgot password? Click here to reset