If you find our articles informative, please follow me to receive updates. It would be even better if you could also follow our ko-fi, where there are many more articles and tutorials that I believe would be very beneficial for you!
如果你觉得我们的文章有料，请关注我获得更新通知，
如果能同时关注我们的 ko-fi 就更好了，
那里有多得多的文章和教程! 相信能使您获益良多.
For collaboration and article reprint inquiries, please send an email to [email protected]
合作和文章转载 请发送邮件至 [email protected]
By: Ash0080
ControlNet Shuffle is a model that has left me feeling quite perplexed, as I find it difficult to define its specific use case. Even after spending several dozen hours exploring various possibilities, I still cannot confidently recommend a fixed optimized workflow because it is just too strange.
The only thing I can share is that you can think of it as a feature mixer that can blend features from a reference image into your target generated image, but this has limitations. In some cases, Shuffle can even produce better image quality than direct generation, making it the only model I have observed that could potentially have a positive impact on image quality. This is unique, as we previously knew that using any ControlNet would only have a negative impact on the image, making Shuffle a special case.
Today, we will use a rather special approach for this lesson. I will showcase some examples of Shuffle's application in several categories to help us better understand its characteristics through comparison.
Feature Mixing
1dog
When we use a photo of a lion to shuffle a picture of a dog, we can see that both the fur color and composition are greatly affected.
People
1girl, fashion,
The images generated by Shuffle are greatly influenced by the original reference image, with significant impact on composition, clothing, color, and background. In fact, one could argue that the generated image is even more "Reference" than the ControlNet Reference . It's worth noting that these results were achieved without using any additional prompts. As I've previously shown in my earlier articles, ControlNet Reference are unable to achieve this without prompts.
best_quality, masterpiece,1girl, fashion,
Words like "best_quality" and "masterpiece" have little to no discernible impact on the images generated by Shuffle.
1girl, fashion,
That's right, the reference image used in Shuffle can have a significant impact on the composition of the generated image. Even without specifying prompts like "close_up", if the reference image is well-matched to the desired content, Shuffle can even enhance the quality of the generated image beyond what would be possible with general image generation. This effect is particularly noticeable when using portrait photography that has been retouched.
1girl, fashion,
So how much feature detail can shuffle actually preserve? When we use a picture of "freckles," we can see that it does not preserve one particular feature, but on the other hand, it does a good job of restoring color and texture features. My guess is that it still determines the weights based on the relative area of the features in the image.
1girl
When I use a reference image that does not match the target content, it blends the color and shape features into the background, and at the same time, we can observe a slight decrease in image quality.
1girl, from_behind
Using a picture with a blurred background, some degree of depth of field and atmospheric feeling is inherited.
1girl,
Using a macro image, we can see that the texture and color of the butterfly are blended into the character's clothing, and the background is blurry but lacks depth.
Architecture
best_quality, masterpiece,
The scene and perspective have been well understood, to the point that it even produces a fisheye lens effect on its own.
best_quality, masterpiece,
When using an image of a concrete building, its sense of emptiness and sharp lines were incorporated into the generated image. I did not write "1girl" in the prompts, so it should be a bias from the model itself. Once again, we can observe the decrease in image quality when the reference image and the target image are different.
best_quality, masterpiece,
Light and shadow, geometry, and color have been recombined.
best_quality, masterpiece,
The Eiffel Tower gives us a steampunk-style image, but the complex texture of the image also leads to a noticeable decrease in image quality.
Pattern
best_quality, masterpiece,
When using fractal patterns or complex designs, my advice is to lower the Control Weight to below 0.4. This usually results in better foreground-background separation. The image above has a Control Weight of 0.8, while the image below has a Control Weight of 0.4.
This is indeed a super convenient way to quickly generate decorative backgrounds, and perhaps in future tutorials, I will explore more applications in this area.
Using high-contrast images can result in unique lighting effects, but it is a double-edged sword and users need to be aware of the decrease in image quality.
best_quality, masterpiece,
On the other hand, images with softer features have less impact on image quality.
Painting
best_quality, masterpiece,
Painting style can also be transferred.
best_quality, masterpiece,
best_quality, masterpiece,
In the end, I tried to see what can be generated using Mona Lisa :D.
Summary
1. Shuffle is a relatively unique model with its feature transfer properties that make its applications potentially infinite. However, if the content of the reference image and the generated image do not match, it can lead to a significant decrease in image quality, which to some extent limits its application.
2. If you plan to use Shuffle for something, my advice is to use the processor to generate image and then put them into ControlNet, and then turn off the preprocessor. Otherwise, when you use hi-res, you may not get the same image because it will shuffle the image again. This comes from my own experience and lesson learned.
3. It is difficult to define the application scope of Shuffle. If you have better experiences or discoveries, please feel free to share them with us.