Using ControlNet to control
Introduction
In this article, i will do fast showcase how to effectively use ControlNet to manipulate poses and concepts. It’s important to note that if you choose to use a different model, you will need to use different ControlNet.
If you don't know what ControlNet is and how to use it with webui i would recommend finding guide for that first.
I will be using Forge webui on this showcase, generally the layout for settings is similar for each webui.
Models used:
CN-anytest_v4-marged_pn_dim256 This model is specifically optimized for pony. There is animagine version of anytest too by same creator. It works to some extend on other XL models too.
The CN-anytest models are a combination of various ControlNet models.
Here is more dept explanation on the anytest models: https://civitai.com/articles/8253/difference-between-anytestv3-and-anytestv4-for-controlnet by mystifying
For this model we do not need preprocessor.
First showcase
1. Reference image
First we need good reference image for what we are trying to do. I am going to use simple example for first showcase:
Fast posing in daz3d and slapping umbrella on photoshop.
Lets see what we can make with this.
2. ControlNet settings
This is where most of the people have problems. What settings to use?
I have highlighted in red what i care about settings:
THESE SETTINGS WORK WELL WITH ANYTEST, might have different results with other models. Understanding what each setting does and learning to adjust them depending on situations is good idea!
We need to have the controlnet enabled obviously.
Select model. If you are using a model that requires a preprocessor, such as the Canny model, make sure to set the correct preprocessor for that model. However, for the CN-anytest model, no preprocessor is needed.
ControlNet weight: You may notice that the ControlNet weight is set above 1. This is intentional, as we want the generation to strongly align with the reference image. A higher weight emphasizes the influence of the reference during the generation process.
Timestep range: We start with a strong influence from the reference image at the beginning of the generation. The influence of ControlNet is gradually reduced, completely stopping at a timestep of 0.4.
Control Mode: In my experience, I prefer the "My prompt is more important" mode. This setting allows the weight of ControlNet to decrease gradually from 1.6 to a lower value, which can help maintain the integrity of the prompt. In contrast, other modes, such as "ControlNet is more important," keep the weight constant at 1.6 throughout the generation, which can overly influence the output.
Resize mode: I recommend using the "Resize and Fill" option to ensure that no parts of the reference image are lost. It’s also beneficial to use a reference image with a similar aspect ratio to your generated output.
What settings should you change if result is bad?
Controlnet weight might need to be lower or higher depending on reference image.
Timestep range might need to be longer if the reference image is more complicated.
You can try testing different control modes, but personally i use "My prompt is more important most of the time.
Next setting might be only on forge
If you have hires enabled it will ask if you want to use the ControlNet for hires. I recommend using only for low res.
3. Rest of the settings
it’s important to provide a clear description of what is happening in your reference image. This helps guide the model in generating the desired output. For the reference image i have i am using:
sitting, holding umbrella, looking up,
Rest of the generation settings and prompt should be what you want.
4. Result
The result is not 1:1 to the reference image, but it is close. Just how i like it.
This was fast example. I might drop more examples using 2 different images, masking etc if there is interest.
Drop comment, like hit the bell and drink coffee or something. (～￣▽￣)～
Now i need to go hide before Novowels finds out i made Elie wet 🏃♀️💨