Inserting Anybody in Diffusion Models via Celeb Basis

by   Ge Yuan, et al.

Exquisite demand exists for customizing the pretrained large text-to-image model, e.g., Stable Diffusion, to generate innovative concepts, such as the users themselves. However, the newly-added concept from previous customization methods often shows weaker combination abilities than the original ones even given several images during training. We thus propose a new personalization method that allows for the seamless integration of a unique individual into the pre-trained diffusion model using just one facial photograph and only 1024 learnable parameters under 3 minutes. So as we can effortlessly generate stunning images of this person in any pose or position, interacting with anyone and doing anything imaginable from text prompts. To achieve this, we first analyze and build a well-defined celeb basis from the embedding space of the pre-trained large text encoder. Then, given one facial photo as the target identity, we generate its own embedding by optimizing the weight of this basis and locking all other parameters. Empowered by the proposed celeb basis, the new identity in our customized model showcases a better concept combination ability than previous personalization methods. Besides, our model can also learn several new identities at once and interact with each other where the previous customization model fails to. The code will be released.


page 7

page 14

page 17

page 18

page 19

page 20

page 21

page 22


Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Text-to-image personalization aims to teach a pre-trained diffusion mode...

InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning

Recent advances in personalized image generation allow a pre-trained tex...

Ablating Concepts in Text-to-Image Diffusion Models

Large-scale text-to-image diffusion models can generate high-fidelity im...

DisenBooth: Disentangled Parameter-Efficient Tuning for Subject-Driven Text-to-Image Generation

Given a small set of images of a specific subject, subject-driven text-t...

Identity Encoder for Personalized Diffusion

Many applications can benefit from personalized image generation models,...

Backdooring Textual Inversion for Concept Censorship

Recent years have witnessed success in AIGC (AI Generated Content). Peop...

Manipulating Embeddings of Stable Diffusion Prompts

Generative text-to-image models such as Stable Diffusion allow users to ...

Please sign up or login with your details

Forgot password? Click here to reset