Introduction.
In this tutorial we're gonna be using ComfyUI, DAMN! v3 as Pony chekpoint and LUSTIFY! v2 as SDXL checkpoint. You can find the workflow file in the attachments.
Pony-based and Illustrious-based checkpoints are known for the excellent prompt understanding and good NSFW capabilities. These models, however, often struggle with realism. Even those Pony/Illustrious models that are finetuned to be realistic, still tend to have a cartoonish aesthetic. This article explores how refining Pony/Illustrious models with SDXL models can create the best of both worlds - highly coherent SFW/NSFW art with the realism of the best SDXL models.
The Refinement Process.
To achieve this, we can use a customized refiner workflow. In this process, Pony/Illustrious models serve as the foundational layer, taking advantage of their strong prompt understanding and NSFW capabilities, while SDXL models are applied slightly later in the diffusion process to enhance the realism.
With the workflow me and @unusual_favor created, the initial gen does that in one press of the button.
Let's get into it!
1. Enter your positive prompts for both Pony/Illustrious (the upper green window) and SDXL (the lower green window). Don't forget that you should be using danbooru-style tags for Pony/Illustrious and something more natural for SDXL. I won't be showcasing any hardcore stuff right in the tutorial, as that would result in a more mature rating of the article, limiting the article's audience. Example of that hardcore stuff (NSFW LINK, don't open at work or in public!): https://civitai.com/posts/5309286
You might be wondering, "what the hell am I looking at?".
Don't worry - it's pretty straightforward once you understand the process. One of the key elements of this workflow is "Seed Base" and "Seed Refine". By changing "Seed Base" you're changing the overall composition. Got a composition you like, but the details from SDXL checkpoint are kinda 'meh'? Change the "Seed Refine" - the composition stays the same while the details are completely new.
Another important aspect is the "Refine At Steps" setting. A good range to start with is 10-13. The later the refinement begins, the more influence the Pony checkpoint will have. Feel free to experiment with this to see what works best for your needs.
You'll also notice a bypassed node called "Latent Interposer." Only unbypass this when you're using Flux instead of Pony, or version 1.5 instead of SDXL. Make sure to set it up according to what you're working with.
The Pony output (on the left) is garbled because it had only 13 steps. That's inteded. Look at the realism of the right pic tho!
"Good. But where do we go from here? I want to upscale the pic, fix the face, etc".
Say no more! You can find the highres fix, tiled upscale and facedetailer nodes lower in the workflow window. By default they're bypassed. You should unbypass them when you've created a pic you like in the first part.
And the good thing is that you have complete control over these parts as well. Don't like the way highres fix looks? Change the seed and/or denoising! Don't like the face "facedetailer" inpainted? Change the seed and/or denoising.
The final output: