If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection

by   Shyamgopal Karthik, et al.

Despite their impressive capabilities, diffusion-based text-to-image (T2I) models can lack faithfulness to the text prompt, where generated images may not contain all the mentioned objects, attributes or relations. To alleviate these issues, recent works proposed post-hoc methods to improve model faithfulness without costly retraining, by modifying how the model utilizes the input prompt. In this work, we take a step back and show that large T2I diffusion models are more faithful than usually assumed, and can generate images faithful to even complex prompts without the need to manipulate the generative process. Based on that, we show how faithfulness can be simply treated as a candidate selection problem instead, and introduce a straightforward pipeline that generates candidate images for a text prompt and picks the best one according to an automatic scoring system that can leverage already existing T2I evaluation metrics. Quantitative comparisons alongside user studies on diverse benchmarks show consistently improved faithfulness over post-hoc enhancement methods, with comparable or lower computational cost. Code is available at <https://github.com/ExplainableML/ImageSelect>.


page 2

page 4

page 9

page 15

page 16


Personalizing Text-to-Image Generation via Aesthetic Gradients

This work proposes aesthetic gradients, a method to personalize a CLIP-c...

FigGen: Text to Scientific Figure Generation

The generative modeling landscape has experienced tremendous growth in r...

Versatile Diffusion: Text, Images and Variations All in One Diffusion Model

The recent advances in diffusion models have set an impressive milestone...

TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition

Text-driven diffusion models have exhibited impressive generative capabi...

Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust

Watermarking the outputs of generative models is a crucial technique for...

Divide and Compose with Score Based Generative Models

While score based generative models, or diffusion models, have found suc...

ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

We present ImageReward – the first general-purpose text-to-image human p...

Please sign up or login with your details

Forgot password? Click here to reset