Sign In
home
models
images
videos
posts
articles
bounties
tools
challenges
events
shop

How to make your checkpoint LoRA friendly?

mixboy's Avatar
mixboy
Aug 28, 2023
resource guide

When using SD1.5 models, you may find that some models can use face LoRA without any problems, while others cannot use face LoRA properly. Let's take a look at the problem and see what are some of the ways to solve this issue.

The first cause of this problem is that most face LoRAs are trained on chilloutmix. LoRAs trained on top of chilloutmix models support chilloutmix models without any problems, but the problem is that face LoRAs don't work well on non-chilloutmix models.

The second reason is that in the case of chilloutmix, the face style is not very individual, so the change from prompt to prompt is quite small, but in a model like BRA v6, the face style is very individual and the change from prompt to prompt is much larger.

A relatively simple approach to the second problem is to use a face embedding such as pureerosface_v1 to change the style of the face to a less distinctive face and use face LoRA on it.

In this post, I will focus on the first reason and discuss how to make an SD model as LoRA friendly as a Chillout Mix.

Brav6

The Brav6 model is an example of a model where face LoRAs do not work as well as intended. What can we do to make face LoRAs work more nicely for this model?

output Blocks

Let's try to fit Brav6 to chilloutmix with a fit minimizer program.

It is known that the output_blocks OUT4 to 07 affect the face a lot. ofcourse, this area affects the face as well as other places, but it's the one that affects the face the most.

To check the overall block level weights we can use a fit program.

(Please see https://github.com/wkpark/sd-model-analyzer fit/model analyzer/merger program)

fit.py chilloutmix_NiPrunedFp16Fix.safetensors Brav6.safetensors --out 4,5,6,7 --alpha 1 -c --fit
...(snip)
 * seed = 114514
output_blocks.4 : 95.7464%
[1.164844]
output_blocks.5 : 96.5678%
[1.110937]
output_blocks.6 : 98.7694%
[1.017187]
output_blocks.7 : 98.5674%
[1.108594]

You can try again with a different seed with an --seed 121212 option.

this result said there is no higher weight problem exists. some checkpoint like blessingMix has a higher weight ratio and it prevents using face LoRA with high weight ratios. but in this case, all block-level weights look not bad.

input blocks

It has been found that input blocks have no higher weights issue.

for IN01,02,04,05 the fit results are follow
 * seed = 114514
input_blocks.1 : 99.7610%
[1.571875]
input_blocks.2 : 99.8744%
[1.239063]
input_blocks.4 : 99.4661%
[1.050781]
input_blocks.5 : 99.6327%
[1.226562]
input_blocks.7 : 97.5953%
[1.082813]
input_blocks.8 : 97.8951%
[1.25625]

But the block-level weights seem strange, especially input_blocks.1

Compared to the chilloutmix, there are imbalances of weights between block levels. In particular, the first block is significantly underweighted.

Replace input_blocks or output_blocks or base encoder

The simplest way to check other possible issues, we can replace input/output blocks or base encoder with a good one.

  • replace base encoder: for BraV6 case, the problem was not fixed

  • replace input_blocks : use chilloutmix's input_blocks. In this case, LoRA works better!!

    • for Brav6, the LoRA issue is mostly caused by input_blocks.

Fix input blocks

Now we know what's the cause of being LoRA-unfriendly. To fix this issue, we can use previous fitting results.

  • Step #1: get block level weights for IN1,2,4,5,7,8 - for example, previous fitting results give us [1.571875, 1.239063, 1.050781, 1.226562, 1.082813, 1.25625]

    • try to fit several times and make optimal weights

  • Step #2: based on step #1 result, merge and make a new checkpoint. in this case, for example, if selected weights are [1.57, 1.24, 1.05, 1.227, 1.083, 1.256] => Add diff [0.57, 0.24, 0.05, 0.227, 0.083, 0.256] for IN01,02,04,05,07,08 blocks.

  • Step #3: check some prompts and compare rendered results.

  • Step #4: try again Step #2 until good results

this is the obtained full input_blocks weight ratio.

(0,0.55,0.55,0.24,0.12,0.05,0.2,0.12,0.08,0.265,0.13,0.08,0.06,0.687,0,0,0,0,0,0,0,0,0,0,0,0)

MERGE1 = Brav6 + (Brav6 - v1-5-pruned-emaonly) x alpha (0.0,0.55,0.55,0.24,0.12,0.05,0.2,0.12,0.08,0.265,0.13,0.08,0.06,0.687,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0)

Fix output blocks

for output blocks, I used dreamshaper_8 to interpolate appearance with BraV6 only as follows:

python fit.py dreamshaper_8.safetensors Brav6.safetensors --out 3,4,5,6,7,8,9,10,11 --fit

and I tried several times and used averaged weight ratios as follows:

(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.13812,1.24609,1.17333,1.13188,1.15336,1.27051,1.05313,0.99458,1.02125)

in this case, I've used a modified fit.py version to merge in the command line prompt:

python fit.py Brav6-fixed-input.fp16.safetensors --out 3,4,5,6,7,8,9,10,11 --alpha 1.13812,1.24609,1.17333,1.13188,1.15336,1.27051,1.05313,0.99458,1.02125 --mode madd -O brav6-dream

Bra v5 fix

Bra V5 has a similar problem. The following result is not yet confirmed, but you can try it and check the results yourself.

v1-5-pruned-emaonly + (braBeautifulRealistic_brav5 - v1-5-pruned-emaonly) x alpha (1.0,1.0,1.21918,1.07896,0.98,0.97875,0.99969,0.98,0.97938,1.00406,1.0,1.0,1.0,1.0,1.0,1.0,1.0,0.99758,1.02188,0.96742,1.02023,1.06891,0.97203,0.88,0.86,0.85945)

Make LoRA

"LoRA = FixedCheckpoint - OriginalCheckpoint" method used to make LoRA, using SuperMerger.

Limitations

  • This LoRA is not perfect. try to change <lora:Brav6fix_v1:1> weight ratio from 0.5~1.5

  • fixed checkpoint works nicely but with less control, but fixer LoRA could control block-level weights using LoRA weight extension more easily.

    • LoRA weights only applied on block-level IN1,2,4,5,7,8 and OUT03~11 (some missing blocks exists)

31

Comments