I tried an experiment that was so stupid that no one would have done it.
There are numerous learning models that support Stable Diffusion. It comes in huge files ranging from 2GB to over 6GB. However, many of you probably know that most of the contents are similar.
The above is an examination of the similarity between Kawaii Animetic V2 and the SD1.5 official model (v1-5-pruned-emaonly.safetensors), and also between KA2 and my another ckpt Blood&Spirit Mix. Compared to the official model, the similarities for each stage of Unet are less than 50% in some cases, but most show similarities between 60 and 80%. It can be said that there is a considerable amount of information wasted. The difference from the BnS Mix (far more general model than v1-5) is approximately 90% or more similar.
This is why I am proposing the concept of Ckpt as a Lora (CaaL). To be more precise, it may be Ckpt as Loras (CaLs).
Concept
The idea is very simple. The SD model is treated as Lora for the reference model. Super Merger has a function to output the difference between two SD models as Lora, so anyone who is extending Super Merger can try it out.
In the following, we will introduce the effectiveness and problems of the Lora conversion of Kawaii Animetic V2 using Dream Shaper 7 as a reference model through the steps involved.
I chose Dream Shaper because it is known by everyone and is not fussy about licenses, it has a medium style that is neither photographic nor animated, and it seems to contain a variety of elements that would make it a good reference model.
How to do?
Now let's open Pandora's box.
Loraizing is easy: just open Super Merger's Lora tab in A1111 (add it if you haven't), specify two ckpts, and press the Make Lora button.
This time, set the file names corresponding to CheckpointA=Kawaii Animetic V2, CheckpointB=Dream Shaper 7. All dimensions are processed with 128 (no).
CheckpointA = Kawaii Animetic V2
CheckpointB = Dream Shaper 7
Lora's output file name is L_DSKA1.
Set Dream Shaper 7 as the ckpt to be used, then go to txt2img to generate the image. The prompt can be anything, but the settings I used this time are as follows.
1girl <lora:L_DSKA1:1>
Steps: 20, Sampler: DPM++ 2M, CFG scale: 6.5, Seed: 68000, Size: 512x768, Clip skip: 2
DS7
DS7+Lora
KA2
The pictures are reasonably close (with a few exceptions), but the fidelity is not quite there. I guess if I merged the two models together, the output would look like this. Is it still impossible to replace ckpt with Lora?
Improve
Is it possible to increase the fidelity even further?
There is also a way to change the size of Lora, but here I tried a multi-stage type. Let's merge the Lora I created this time into DS7, make it a little closer to KA2, and then take further differences. I created model C_DSKA1 by merging the created Lora (L_DSKA1) into DS7, and created the difference Lora (L_DSKA2) between KA2 and DS7K1.
L_DSKA1 = KA2 - DS7
C_DSKA1 = DS7 + L_DSKA1
L_DSKA2 = KA2 - C_DSKA1
I further created and synthesized up to L_DSKA4 using the same way.
Here are the results (I don't know why DS7 and DS7+Lora images are different from above).
The second image clearly shows an improvement in fidelity.
There was almost no change from 3 to 4 steps. It seems that what is close is close, but what is different is different. Nevertheless, it is fair to say that the general characteristics of the models are now fairly well reproduced, except that the outputs are not "exactly the same".
Another prompt
The big problem with this sample is that it is clearly burnt. This may not be practical.
However, this setting was also causing burns in KA2, so I readjusted the parameters.
...... But I can't upload pictures any more. Then this article continues to the 2nd part.