## PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding [](https://huggingface.co/papers/2312.04461)
[[Paper](https://huggingface.co/papers/2312.04461)] [[Project Page](https://photo-maker.github.io)] [[Model Card](https://huggingface.co/TencentARC/PhotoMaker)]
[[💥New 🤗 **Demo (PhotoMaker V2)**](https://huggingface.co/spaces/TencentARC/PhotoMaker-V2)] [[🤗 Demo (Realistic)](https://huggingface.co/spaces/TencentARC/PhotoMaker)] [[🤗 Demo (Stylization)](https://huggingface.co/spaces/TencentARC/PhotoMaker-Style)]
[[Replicate Demo (Realistic)](https://replicate.com/jd7h/photomaker)] [[Replicate Demo (Stylization)](https://replicate.com/yorickvp/photomaker-style)] [[Jittor version](https://github.com/NK-JittorCV/nk-diffusion)]
**PhotoMaker-V2** is supported by the [HunyuanDiT](https://github.com/Tencent/HunyuanDiT) team.
🥳 We release **PhotoMaker V2**. Please refer to [comparisons](./README_pmv2.md) between PhotoMaker V1, PhotoMaker V2, IP-Adapter-FaceID-plus-V2, and InstantID. Please watch [this video](https://photo-maker.github.io/assets/demo_pm_v2_full.mp4) for how to use our demo. For PhotoMaker V2 ComfyUI nodes, please refer to the [Related Resources](https://github.com/TencentARC/PhotoMaker?tab=readme-ov-file#related-resources)
## 🚩 **New Features/Updates**
- ✅ July 22, 2024. 💥 We release PhotoMaker V2 with **improved ID fidelity**. At the same time, it still maintains the generation quality, editability, and compatibility with any plugins that PhotoMaker V1 offers. We have also provided scripts for integration with [ControlNet](./inference_scripts/inference_pmv2_contronet.py), [T2I-Adapter](./inference_scripts/inference_pmv2_t2i_adapter.py), and [IP-Adapter](./inference_scripts/inference_pmv2_ip_adapter.py) to offer excellent control capabilities. Users can further customize scripts for upgrades, such as combining with LCM for acceleration or integrating with IP-Adapter-FaceID or InstantID to further improve ID fidelity. We will release technical report of PhotoMaker V2 soon. Please refer to [this doc](./README_pmv2.md) for a quick preview.
- ✅ January 20, 2024. An **important** note: For those GPUs that do not support bfloat16, please change [this line](https://github.com/TencentARC/PhotoMaker/blob/6ec44fc13909d64a65c635b9e3b6f238eb1de9fe/gradio_demo/app.py#L39) to `torch_dtype = torch.float16`, the speed will be **greatly improved** (1min/img (before) vs. 14s/img (after) on V100). The minimum GPU memory requirement for PhotoMaker is **11G** (Please refer to [this link](https://github.com/TencentARC/PhotoMaker/discussions/114) for saving GPU memory).
- ✅ January 15, 2024. We release PhotoMaker.
---
## 🔥 **Examples**
### Realistic generation
- [](https://huggingface.co/spaces/TencentARC/PhotoMaker)
- [**PhotoMaker notebook demo**](photomaker_demo.ipynb)
## PhotoMaker V2: Improved ID Fidelity and Better Controllability Compared to PhotoMaker V1
[[🤗 Demo](https://huggingface.co/spaces/TencentARC/PhotoMaker-V2)]
When training PhotoMaker V2, we focused on improving ID fidelity. Compared to PhotoMaker V1, we introduced 1️⃣ new training strategies, incorporated 2️⃣ more portrait datasets, and utilized 3️⃣ a more powerful ID extraction encoder. We will release a technical report soon. Thank you all for your attention.
### 🌠 **Key improvements in PhotoMaker V2:**
1. **ID fidelity** has been **further improved**, especially for single image input and Asian facial inputs. Of course, feeding more facial images can still yield better results.
2. By integrating [ControlNet](./inference_scripts/inference_pmv2_contronet.py), [T2I-Adapter](./inference_scripts/inference_pmv2_t2i_adapter.py), and [IP-Adapter](./inference_scripts/inference_pmv2_ip_adapter.py), the generation process becomes **more controllable**. We provide corresponding scripts for reference. Additionally, PhotoMaker V2 allows users to achieve better ID consistency by combining it with IP-Adapter-FaceID, InstantID, and [character LoRA](https://github.com/TencentARC/PhotoMaker/discussions/14).
3. PhotoMaker V2 **inherits the promising features of PhotoMaker V1**, such as high-quality and diverse generation capabilities, and powerful text control. Additionally, it can still integrate previous applications like bringing characters from old photos or paintings back to reality, identity mixing, and changing age or gender.
## Comparisons with PhotoMaker V1, IP-Adapter-FaceID and InstantID
We selected the three most prevalent methods in ID personalization generation, namely PhotoMaker V1, [IP-Adapter-FaceID-Plus-V2](https://huggingface.co/h94/IP-Adapter-FaceID) ([best of IP-Adapter-FaceID](https://github.com/cubiq/ComfyUI_IPAdapter_plus/issues/195)), and [InstantID](https://github.com/InstantID/InstantID).
To ensure a fair comparison, we used the same base model ([RealVisXL-V4.0](https://huggingface.co/SG161222/RealVisXL_V4.0)) and scheduler ([Euler](https://huggingface.co/docs/diffusers/api/schedulers/euler)), and selected the best out of four randomly generated images from each method for visualization. The prompts and negative prompts were consistent:
Prompt: `instagram photo, portrait photo of a woman img holding two cats, colorful, perfect face, natural skin, hard shadows, film grain`
Negative Prompt: `(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth`
We can see that our method has **advantages** in maintaining ID fidelity and in the quality of the generated images
## Cooperation with ControlNet / T2I-Adapter / IP-Adapter
PhotoMaker V2 can collaborate with [T2I-Adapter’s doodle mode](https://huggingface.co/TencentARC/t2i-adapter-sketch-sdxl-1.0),
allowing for controlled image generation based on user drawings and prompts.
This feature can be experienced in [[🤗 our official demo]](https://huggingface.co/spaces/TencentARC/PhotoMaker-V2).
The following video is an example of the experience process:
https://github.com/user-attachments/assets/1303d684-89e4-49d2-8e8c-4b659c8b48e7
Additionally, PhotoMaker V2 can work with [ControlNet](https://github.com/lllyasviel/ControlNet) and [T2I-Adapter](https://github.com/TencentARC/T2I-Adapter) for layout control, such as edge, pose, depth, and more.
We provide two example scripts:
1. [inference_pmv2_contronet.py](./inference_scripts/inference_pmv2_contronet.py)
2. [inference_pmv2_t2i_adapter.py](./inference_scripts/inference_pmv2_t2i_adapter.py)
The image below is an example of controlled generation using pose through ControlNet:
Our sample scripts can be referred to:
[inference_pmv2_ip_adapter.py](./inference_scripts/inference_pmv2_ip_adapter.py)
The image below is an example:
PhotoMaker V2, as a plugin, can work well with other plugins, such as IP-Adapter-FaceID or InstantID, to further improve ID fidelity, or combining with LCM for acceleration. We look forward to your exploration of more features, and welcome you to **provide PRs** or **contribute to the open-source community**
🥳 If you have built or known repositories or applications around PhotoMaker V2, please leave us a message in the discussion. We will include them in our README.
## LICENSE
Since PhotoMaker V2 relies on [InsightFace](https://github.com/deepinsight/insightface), it also needs to comply with its [license](https://github.com/deepinsight/insightface?tab=readme-ov-file#license).
================================================
FILE: cog.yaml
================================================
# Configuration for Cog ⚙️
# Reference: https://github.com/replicate/cog/blob/main/docs/yaml.md
build:
# set to true if your model requires a GPU
gpu: true
# python version in the form '3.11' or '3.11.4'
python_version: "3.11"
python_packages:
- "accelerate==0.26.1"
- "diffusers==0.25.0"
- "huggingface-hub==0.20.2"
- "numpy==1.24.4"
- "omegaconf==2.3.0"
- "peft==0.7.1"
- "safetensors==0.4.1"
- "torch==2.1.1"
- "torchvision==0.16.1"
- "transformers==4.36.2"
run:
- curl -o /usr/local/bin/pget -L "https://github.com/replicate/pget/releases/download/v0.5.6/pget_linux_x86_64" && chmod +x /usr/local/bin/pget
# predict.py defines how predictions are run on your model
predict: "predict.py:Predictor"
image: r8.im/tencentarc/photomaker
================================================
FILE: feature-extractor/preprocessor_config.json
================================================
{
"crop_size": 224,
"do_center_crop": true,
"do_convert_rgb": true,
"do_normalize": true,
"do_resize": true,
"feature_extractor_type": "CLIPFeatureExtractor",
"image_mean": [
0.48145466,
0.4578275,
0.40821073
],
"image_std": [
0.26862954,
0.26130258,
0.27577711
],
"resample": 3,
"size": 224
}
================================================
FILE: gradio_demo/app.py
================================================
import torch
import numpy as np
import random
import os
import sys
from diffusers.utils import load_image
from diffusers import EulerDiscreteScheduler
from huggingface_hub import hf_hub_download
import spaces
import gradio as gr
from photomaker import PhotoMakerStableDiffusionXLPipeline
from style_template import styles
from aspect_ratio_template import aspect_ratios
# global variable
base_model_path = 'SG161222/RealVisXL_V4.0'
try:
if torch.cuda.is_available():
device = "cuda"
elif sys.platform == "darwin" and torch.backends.mps.is_available():
device = "mps"
else:
device = "cpu"
except:
device = "cpu"
MAX_SEED = np.iinfo(np.int32).max
STYLE_NAMES = list(styles.keys())
DEFAULT_STYLE_NAME = "Photographic (Default)"
ASPECT_RATIO_LABELS = list(aspect_ratios)
DEFAULT_ASPECT_RATIO = ASPECT_RATIO_LABELS[0]
# download PhotoMaker checkpoint to cache
photomaker_ckpt = hf_hub_download(repo_id="TencentARC/PhotoMaker", filename="photomaker-v1.bin", repo_type="model")
torch_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
if device == "mps":
torch_dtype = torch.float16
pipe = PhotoMakerStableDiffusionXLPipeline.from_pretrained(
base_model_path,
torch_dtype=torch_dtype,
use_safetensors=True,
variant="fp16",
# local_files_only=True,
).to(device)
pipe.load_photomaker_adapter(
os.path.dirname(photomaker_ckpt),
subfolder="",
weight_name=os.path.basename(photomaker_ckpt),
trigger_word="img",
pm_version="v1",
)
pipe.id_encoder.to(device)
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)
# pipe.set_adapters(["photomaker"], adapter_weights=[1.0])
pipe.fuse_lora()
pipe.to(device)
@spaces.GPU
def generate_image(upload_images, prompt, negative_prompt, aspect_ratio_name, style_name, num_steps, style_strength_ratio, num_outputs, guidance_scale, seed, progress=gr.Progress(track_tqdm=True)):
# check the trigger word
image_token_id = pipe.tokenizer.convert_tokens_to_ids(pipe.trigger_word)
input_ids = pipe.tokenizer.encode(prompt)
if image_token_id not in input_ids:
raise gr.Error(f"Cannot find the trigger word '{pipe.trigger_word}' in text prompt! Please refer to step 2️⃣")
if input_ids.count(image_token_id) > 1:
raise gr.Error(f"Cannot use multiple trigger words '{pipe.trigger_word}' in text prompt!")
# determine output dimensions by the aspect ratio
output_w, output_h = aspect_ratios[aspect_ratio_name]
print(f"[Debug] Generate image using aspect ratio [{aspect_ratio_name}] => {output_w} x {output_h}")
# apply the style template
prompt, negative_prompt = apply_style(style_name, prompt, negative_prompt)
if upload_images is None:
raise gr.Error(f"Cannot find any input face image! Please refer to step 1️⃣")
input_id_images = []
for img in upload_images:
input_id_images.append(load_image(img))
generator = torch.Generator(device=device).manual_seed(seed)
print("Start inference...")
print(f"[Debug] Prompt: {prompt}, \n[Debug] Neg Prompt: {negative_prompt}")
start_merge_step = int(float(style_strength_ratio) / 100 * num_steps)
if start_merge_step > 30:
start_merge_step = 30
print(start_merge_step)
images = pipe(
prompt=prompt,
width=output_w,
height=output_h,
input_id_images=input_id_images,
negative_prompt=negative_prompt,
num_images_per_prompt=num_outputs,
num_inference_steps=num_steps,
start_merge_step=start_merge_step,
generator=generator,
guidance_scale=guidance_scale,
).images
return images, gr.update(visible=True)
def swap_to_gallery(images):
return gr.update(value=images, visible=True), gr.update(visible=True), gr.update(visible=False)
def upload_example_to_gallery(images, prompt, style, negative_prompt):
return gr.update(value=images, visible=True), gr.update(visible=True), gr.update(visible=False)
def remove_back_to_files():
return gr.update(visible=False), gr.update(visible=False), gr.update(visible=True)
def remove_tips():
return gr.update(visible=False)
def randomize_seed_fn(seed: int, randomize_seed: bool) -> int:
if randomize_seed:
seed = random.randint(0, MAX_SEED)
return seed
def apply_style(style_name: str, positive: str, negative: str = ""):
p, n = styles.get(style_name, styles[DEFAULT_STYLE_NAME])
return p.replace("{prompt}", positive), n + ' ' + negative
def get_image_path_list(folder_name):
image_basename_list = os.listdir(folder_name)
image_path_list = sorted([os.path.join(folder_name, basename) for basename in image_basename_list])
return image_path_list
def get_example():
case = [
[
get_image_path_list('./examples/scarletthead_woman'),
"instagram photo, portrait photo of a woman img, colorful, perfect face, natural skin, hard shadows, film grain",
"(No style)",
"(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth",
],
[
get_image_path_list('./examples/newton_man'),
"sci-fi, closeup portrait photo of a man img wearing the sunglasses in Iron man suit, face, slim body, high quality, film grain",
"(No style)",
"(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth",
],
]
return case
### Description and style
logo = r"""
