Controlnet for DensePose

This Controlnet model accepts DensePose annotation as input

How to use

Put the .safetensors file under ../stable diffusion/models/ControlNet/

About DensePose

This repo explains WHAT is DensePose and HOW to use it: https://github.com/facebookresearch/detectron2/tree/main/projects/DensePose

To use the annotator, you need to install Detectron2 as instructed. The DensePose annotator relies on apply_net.py to run. Please refer to the last image in the post for necessary modification of this script.

Transforming this project into a preprocessor turns out to be too challenging for me.

Model Feature

Strength

1. Stable body pose

2. Good performance on inferring hands

3. Ability to infer tricky poses

4. Great potential with Depth Controlnet

Weakness

1. Like Openpose, depth information relies heavily on inference and Depth Controlnet

2. Unstable direction of head

3. Intention to infer multiple person (or more precisely, heads)

Issues that you may encouter

Q: This model tends to infer multiple person

A: Avoid leaving too much empty space on your annotation. Or use it with depth Controlnet. Or write the prompt as what I did (eg. (one:1.2) girl).

Q: This model doesn't perform well

A:

1. The DensePose annotator doesn't work well. You can mannually modify the annotation. It should be easier than modifying an OpenPose

2. The depth Annotator doesn't work well. In fact, I recommend to use glpn-nyu as the depth annotator. Please refer to https://huggingface.co/vinvino02/glpn-nyu

3. The pose is too tricky. It goes beyonds the model's ability.

Q: This model doesn't perform well with my LoRA

A:

1. That probably means your LoRA is not trained on enough data. It turns out that LoRA trained on enough amount of data will have fewer conflicts with Controlnet or your prompts

2. Change your LoRA IN block weights to 0. It's always the IN block that causes all the conflicts. Please refer to https://github.com/hako-mikan/sd-webui-lora-block-weight

*Note: The second pose generation in the post uses depth Controlnet (w=0.25)