I have been working on two separate checkpoint now:
Codename HoJ: a more semi-realistic checkpoint, the one i used to extract Cocktail Juice
Codename UaK: a more "toon" like checkpoint, with exaggerated proportions, leveraging QuirkFix-PlusV3 i used to build QuirkV2
A basic merge of those checkpoint give an interesting result but i feel those two checkpoints are the perfect test subjects for Block merging.
For the few of those not knowing, an SDXL model is composed of several "blocks". Basically, they are a few layers of neuron which put together achieve a specific function.
There is "BASE", "IN00" to "IN08", "MID" and "OUT08" to "OUT00". And when merging models with more advanced tools like Supermerger, you can basically choose the rate of merge of each block instead of using the same ratio for all.
But what would be the value to put in each of those? Each of those block are trained during model training and depending on the base model and how it was done, the "impact" of each of those block may differ (but they probably still have a similar role as in base SDXL if the model has been fine-tuned from it).
This old article is a very good starting point, but i want to figure it out myself. (I put the included picture from the article here in case it disappear in the future.
So, i'll start a script to create a "all or nothing" merge of both these models, block by block, one by one, generate the same image for test and try to put the results in a standalone page to check the effect more easily. That's only 20 images if i change only one block at a time.... but 1 048 576 pictures if try for all the combination 😱
Even at 30s per images, it would take a year to generate all of those, as much as i like playing with these models, that's too much even for me ha ha :D
Thanks for reading! (and if among the alpha values in the for grid you have a preference, feel free to comment!)
Edit: I figured out how to provide a list of Merge Block Weights for an XYZ plot in Supermerger :D
EDIT 2: I feel something is not right in most merge tutorial/experiment i saw. For that, let's go back quickly to SDXL UNET
In this classic diagram (that is for SD 1.x, but still fit), i have removed the less import annotations to keep the important part: The blocks and the "?!?" at the top (originally, "Concatenate").
Indeed, the picture is generated by going through all the blocks in order, but IN blocks also send information to the corresponding OUT blocks. So, i feel that if we merge an IN or OUT block with a specific weight, it would yield better result to do the same with the corresponding pair block.
But which is the correct number? Are they going 0 through 8 for both type in this order? Or as the Supermerger UI suggest, it goes 0 to 8 for IN, and the 8 to 0 for OUT? (which would be more logical, IN02 is connected then to OUT02).
I couldn't find any clear documentation regarding that, so... i went and read some code from Stable-Diffusion.cpp: it seems the same index is used for Output blocks and Input blocks when the data from the Input block is used in the Output block (so, IN00 with OUT00 and so on...)
I'll have to test that too :D