Hello all. Today I wanted to discuss a dirty secret that I've been using for a while. Many may already know about this, but there's a likelihood many of you fine folks also don't. The trick? Well, it's not just refining, but something a bit more that may explode your creative abilities.
WHAT IS REFINING, AND WHY SHOULD I CARE?
Traditionally, refining allowed artists to add further details to an image, or rather, 'refine' them. This allows an artist to take the best qualities of two different checkpoints, made for this intended purpose.
Base Checkpoint: Again, traditionally, a standard model or checkpoint, such as SDXL (Stable Diffusion XL) that generates base images.
Refiner Checkpoint: Originally, a model or checkpoint designed to refine a model's image, denoise, etc., such as SDXL Refiner.
As base checkpoints have gotten better in output, and resolutions increased, the use of refining has become less common. One of the strengths of refining, however, was the ability to use two checkpoints at the same time.
What does this mean? Simply, you can take the strengths of two different checkpoints, and combine them much like model merging, without all the nasty work. Let's continue, because this is very important.
GENERAL OBSERVATIONS:
I find that anime checkpoints, as we'll show, produce very high quality outputs for vivid color, great poses, and cinematic camera angles. Oppositionally, photoreal checkpoints tend to crank the detail with far more unique faces. Not all checkpoints adhere to this observation, but generally speaking, it's part of the strengths. 2.5D checkpoints bridge that gap, but I often find they have a 'waxiness' that I personally find off-putting, even if the rest of it is immaculate.
Well, since we now know we can essentially use refining to combine checkpoints, now we have the ability to utilize the strengths of various checkpoints, and combine them together. This also means we can focus on various art styles, or other attributes that checkpoints exhibit. In essence, we just slammed open the door to ideas, and can go hog wild!
The possibilities now are endless, but I need to show you why.
CHECKPOINTS USED:
Obviously, this isn't exhaustive, but I've taken three impressive Illustrious checkpoints, and will refine/mix them variably, and post the results. These are checkpoints I've tested before, and make for a good baseline.
One obsession 1.3 2.5D - Illustrious Base
Download Link: -> HERE
Author: maxfeifei8
Purpose of Testing: Comparative checkpoint for quasi realistic anime styles
Style: Anime / qausi-realistic with emphasis on anime
Pros: Strong anime influence, wonderful poses, great backgrounds, great expressions, beautiful stunning images, brand spanking new checkpoint (as of this post)
Cons: Has trouble inpainting occasionally, doesn't play well with some LoRAs and refiners, has a mind of it's own at times, somewhat low actor variability requires more prompting
ToonMerge Illustrious V5.0 Semi - Illustrious Base
Download Link: -> HERE
Author: Pinnacle
Purpose of Testing: Baseline (this is a workhorse checkpoint that I always come back to)
Style: 2.5D / semi-realistic anime
Pros: Strong prompt following, accurate, creative, strong workhorse, high variety
Cons: Sometimes not as creative as other checkpoints, sometimes require reinforcement prompting
CyberIllustrious | CyberRealistic v4.0 - Illustrious Base
Download Link: -> HERE
Author: Cyberdelia
Purpose of Testing: Comparative checkpoint for more popular and realistic styles
Style: Realistic / Photorealism
Pros: Very strong photorealism, creative within context of realism, high variety of faces/actors, inpaints extremely well, excellent as base or refiner, extremely pliable with LoRAs and other checkpoints
Cons: Most of the cons are related to problems inherent with realistic checkpoints such as lack of creativity, poses, etc., some deformation requires inpainting, REALLY likes to be naked (naughty checkpoint)
TESTING PARAMETERS:
All images generated as one-shots, with no alteration, 5 at a time. If an image is NSFW, kept generating until next 'safe' image.
CFG: 4 on all checkpoints
Steps: 30 on all checkpoints
Positive Prompt: incredibly absurdres, highres, masterpiece, dark fantasy, newest, great angle, (abstract watercolor background), woman posing with arms up, adventurer attire, magic, vivid
Negative Prompt: worst quality, bad quality, low quality, lowres
Sampler: Euler A on all checkpoints
VAE: None on all checkpoints (except what is built into the checkpoint itself)
Image Resolution: 832x1216 (reduced to 541x790 to keep quality, then stitched together in 2704x790 in GIMP. Images were not rendered in 541x790 due to quality issues of output)
Number of Images Produced: 5 per test, then stitched together.
LoRAs Used: None on all checkpoints.
Seed: 6943948851721725385
Baseline 100% Checkpoint - No Refiner
The checkpoints were ran as, 'as is,' with no refinement. The purpose was to show how they work by themselves.
One obsession 1.3 2.5D
ToonMerge Illustrious V5.0 Semi
CyberIllustrious | CyberRealistic v4.0
Excellent, now we've established a baseline.
Notes:
- One obsession tends to be the most creative in angles and poses, with great color variety but lacks actor variety.
- ToonmMerge represents a good middle ground between both checkpoints, with a touch more detail, more actor variety, and more grounded backgrounds, although less color variation and abstract imagery.
- CyberIllustrious cranks up the detail, but loses some of the great poses and camera angles. Does surprisingly well with abstract art, however.
Checkpoint Refinement - (~66/33) 67% Base Checkpoint w/ 33% CyberIllustrious as Refiner
One obsession 1.3 2.5D
ToonMerge Illustrious V5.0 Semi
Notes:
- Slight shift on faces to be a tad more realistic, and more variety across the board, while retaining more anime aesthetics.
- Anime checkpoints become more 2.5D, without losing poses and camera angles.
- Slightly more detail injected into images.
Checkpoint Refinement - (~33/67) 33% Base Checkpoint w/ 67% CyberIllustrious as Refiner
One obsession 1.3 2.5D
ToonMerge Illustrious V5.0 Semi
Notes:
- Drastic shift towards photorealism, while maintaining significant poses, and camera angles.
- Body shapes maintain curviness, as is popular with anime checkpoints.
- Again, more detail injected into image.
Checkpoint Refinement - (~67/33) 33% Base Checkpoint w/ 33% One obsession as Refiner
ToonMerge Illustrious V5.0 Semi
CyberIllustrious | CyberRealistic v4.0
Notes:
- ToonMerge shows a bit more color depth and maintains almost all original qualities
- CyberIllustrious maintains more abstract nature
- Only slight reduction in detail
- Faces and actors shifted more towards anime or 2.5D styles
Checkpoint Refinement - (~33/67) 33% Base Checkpoint w/ 67% One obsession as Refiner
ToonMerge Illustrious V5.0 Semi
CyberIllustrious | CyberRealistic v4.0
Notes:
- ToonMerge gains significant color variation
- CyberIllustrious goes wild with the abstract artistry, and added curviness
- Drastic shift towards anime actors, although less facial variation
- A reduction in background detail
_______________________________________________
At this point I stopped the test, as I don't think we'd see as drastic changes using ToonMerge as a refiner, since it tends to bridge the gap between the two checkpoints styles. Of note, I tend to use Toon as both a refiner and a base, depending on what I'm working on - it's one of the strengths of this model. It doesn't give drastic tonal shifts, but maintains a steady middle of the line cadence for the work I like to do regarding photorealism and anime/cartoon styles.
FINAL NOTES:
- Mixing checkpoints, both as refiners and bases, allows you to extract the qualities of both
- Heavily artistic/colorful checkpoints can bring a lot to the table and liven up a scene
- Anime/cartoon checkpoints can give better angles and curviness
- Photorealistic checkpoints bring detail
- All checkpoints have different qualities, not just what is listed above
- Some checkpoints may be overall better at producing better art styles, creatures, and backgrounds, and this allows you mix them
As you can see, mixing checkpoints can produce wildly different results, and really improve the overall quality of your images, depending on what you're trying to accomplish. I've used this to great effect, in refining, and img2img, to really bring out specific qualities, while retaining other aspects. If nothing else, I encourage everyone to give this a try with their favorite checkpoints. The more artistic and wild, the crazier the results will get.
Of note, different base models can produce very poor results with other base models - for example, using a Pony checkpoint and refining with an Illustrious checkpoint. While I've had some success refining Illustrious models with Pony models, I haven't had nearly as much success the other way around, often ending up with blown images. Results will vary greatly, and I encouraged lots of playing around and testing.
_______________________________________________
I hope this opens up a lot of doors for people to try different things. It's one of the greatest tricks I learned, which I use in almost every generation. It's what led to Pzeudo being made by epiTune.
Warmest regards,
Bjorn