DiffGAR: Model-Agnostic Restoration from Generative Artifacts Using Image-to-Image Diffusion Models

by   Yueqin Yin, et al.

Recent generative models show impressive results in photo-realistic image generation. However, artifacts often inevitably appear in the generated results, leading to downgraded user experience and reduced performance in downstream tasks. This work aims to develop a plugin post-processing module for diverse generative models, which can faithfully restore images from diverse generative artifacts. This is challenging because: (1) Unlike traditional degradation patterns, generative artifacts are non-linear and the transformation function is highly complex. (2) There are no readily available artifact-image pairs. (3) Different from model-specific anti-artifact methods, a model-agnostic framework views the generator as a black-box machine and has no access to the architecture details. In this work, we first design a group of mechanisms to simulate generative artifacts of popular generators (i.e., GANs, autoregressive models, and diffusion models), given real images. Second, we implement the model-agnostic anti-artifact framework as an image-to-image diffusion model, due to its advantage in generation quality and capacity. Finally, we design a conditioning scheme for the diffusion model to enable both blind and non-blind image restoration. A guidance parameter is also introduced to allow for a trade-off between restoration accuracy and image quality. Extensive experiments show that our method significantly outperforms previous approaches on the proposed datasets and real-world artifact images.


page 1

page 6

page 11


DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior

We present DiffBIR, which leverages pretrained text-to-image diffusion m...

Generative Diffusion Prior for Unified Image Restoration and Enhancement

Existing image restoration methods mostly leverage the posterior distrib...

BIGRoC: Boosting Image Generation via a Robust Classifier

The interest of the machine learning community in image synthesis has gr...

A Unified Conditional Framework for Diffusion-based Image Restoration

Diffusion Probabilistic Models (DPMs) have recently shown remarkable per...

Artifact Restoration in Histology Images with Diffusion Probabilistic Models

Histological whole slide images (WSIs) can be usually compromised by art...

My Art My Choice: Adversarial Protection Against Unruly AI

Generative AI is on the rise, enabling everyone to produce realistic con...

PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance

Exploiting pre-trained diffusion models for restoration has recently bec...

Please sign up or login with your details

Forgot password? Click here to reset