Crody's Model Merge Guide

Hi this is Crody from Team-C: creator of Nova Series

In this article, I'll write down what kind of merge I use with some knowledge about SDXL models
For how I do, please read Merge Scripter Guide first

1. Weighted Sum / Sum Twice

Weighted Sum (WS) merges 2 models, Sum Twice (ST) merges first 2 and 1 model (which means doing WS twice)
You can use Block Merge as well
Using alpha (and beta) to determine how much similarity the result have
Higher value means the results would be similar to latter model

MMS Text:

WS

CM A + B alpha Result

A = 1 - alpha
B = alpha

ST

CM A + B +S C alpha beta Result

A = (1 - alpha) * (1 - beta)
B = alpha * (1 - beta)
C = alpha * beta

Triple Sum (TRS) Merges 3 models with independent ratios

CM A + B +T C alpha beta Result

A = 1 - (alpha + beta)
B = alpha
C = beta

2. Add Difference

Add Difference (AD) merges the difference between latter 2 models into the first model
This can be used when the model needs additional information

MMS Text:

CM A + B - C alpha Result

A + ((B - C) * alpha)

3. Block Merge

The way to handle values independently for each parts
I'll write down what kind of value it has on below

Overall analysis:
BASE : text encoder
IN00 - IN05 : handles anatomy and details
IN06 - IN08 : handles tone and poses
MID00 : over all similarity
OUT00 - OUT03 : handles shape of face and body
OUT04 - OUT08 : handles textures

• BASE: text encoder, handles what kind of system you use
  This also handles the prompt encoding :
  if you merge SDXL 1.0 models and Pony models with 0.3 BASE, it results in gray images or worse results (using DARE merge also leads in same result so I recommend to set it 0 or 1)
  if the base structure is same (Illustrious and NoobAI shares same structure), the BASE can be in between 0 to 1 using DARE merge
  

• IN00: handles overall inputs, outlines, contrast and fingers
  To avoid finger errors in Illustrious models, I don't recommend float value (other than 0 and 1) for this as well
  

• IN01: handles background, depth of view and some details in hair and clothes
  

• IN02 and IN03: handles anatomy details, hair and clothes
  

• IN04 and IN05: handles textures, body anatomy and behavior for NSFW prompts
  

• IN06: handles clothing texture and details
  

• IN07: handles pose structures
  

• IN08: handles texture feels, depth and anatomy base
  

• MID00: handles how much it looks like to the latter model
  This also handles prompt adherence and drawing style as well
  

• OUT00: handles torso, face structure, age and eyes
  

• OUT01: handles upper body and face shape
  

• OUT02: handles body curves, nose and ears
  

• OUT03: handles skin color and mouth
  

• OUT04 and OUT05: handles overall texture and smooth lights
  

• OUT06: handles overall texture and hard lights
  

• OUT07: handles overall colors and rough draw lines
  

• OUT08: handles output, colors and drawing lines

MMS Text:

Replace the alpha (beta) with following

"BASE, IN00, IN01, IN02, IN03, IN04, IN05, IN06, IN07, IN08, MID00, OUT00, OUT01, OUT02, OUT03, OUT04, OUT05, OUT06, OUT07, OUT08"

Replacing each named block with float value in 0 - 1 do the job
For example, if you want to merge 2 models using WS with 0.4 for posing and facial structure, model likeness with 0.3 and other is 0, you can do the following

CM A + B "0,0,0,0,0,0,0,0.4,0.4,0.4,0.3,0.4,0.4,0.4,0.4,0,0,0,0,0" Result

4. Fine Tuning

Handles details, contrast, brightness and colors with using 8 values

1. Detail-IN (default:1)

2. Detail-MID (default:1)

3. Detail-OUT (default:1)

4. Contrast (default:1)

5. Brightness (default:1)

6. Red/Cyan (default:0)

7. Green/Magenta (default:0)

8. Blue/Yellow (default:0)

For detail values (1-3), make the sum of them around 3.0 and highest value in 1.3
If you have higher contrast, make the brightness high as well
If you change the detail values, please block merge model's IN00 afterwards to maintain the finger details

Notebook edit (MMS cannot determine finetune value):

...
--fine "DtIN,DtMID,DtOUT,CONT,BRIG,RED,GREEN,BLUE \ 
...

Replace each value with float would do the job

5. Example Merge using techniques above

MMS text:

+NA, https://civitai.com/models/376130?modelVersionId=1474209 
+NO, https://civitai.com/models/967405/nova-orange-xl 
+NU, https://civitai.com/models/982047?modelVersionId=1445113

CM NA + NO +S NU "0,1,1,0.2,0.2,0.3,0.3,0,0,0,0,0.2,0.1,0.1,0.1,0.4,0.4,0.4,0.4,0.4" "0,0,0,0.3,0.3,0.2,0.2,0.3,0,0,0,0.1,0.1,0.1,0.2,0.2,0.2,0.2,0.2,0.2" _A
-NA
-NU

CM _A + NO "0,1,0,0,0,0,0,0,0.2,0.2,0,0.3,0,0,0,0,0,0,0,0" TestResult

Notebook edit: (add fine tune)

... 
!python merge.py "ST" "/kaggle/tmp/models/" "NA.safetensors" "NO.safetensors" --model_2 "NU.safetensors" \ 
--vae "/kaggle/tmp/vae/VAE.safetensors" \  
--alpha "0,1,1,0.2,0.2,0.3,0.3,0,0,0,0,0.2,0.1,0.1,0.1,0.4,0.4,0.4,0.4,0.4" \  
--beta "0,0,0,0.3,0.3,0.2,0.2,0.3,0,0,0,0.1,0.1,0.1,0.2,0.2,0.2,0.2,0.2,0.2" \  
--fine "1.1,1,0.9,1.2,1.1,0,0,0 \ 
--save_half --prune --save_safetensors --output "TEMP_A"  
flush() 

TEMP_A = model("TEMP_A",1)

remove_model(NA)
remove_model(NU)
flush()

!python merge.py "WS" "/kaggle/tmp/models/" "TEMP_A.safetensors" "NO.safetensors" \
--vae "/kaggle/tmp/vae/VAE.safetensors" \  
--alpha "0,1,0,0,0,0,0,0,0.2,0.2,0,0.3,0,0,0,0,0,0,0,0" \
--save_half --prune --save_safetensors --output "TestResult"  
flush() 
...  

6. Bonus: Nova Furry XL Illustrious v5.0 Merge

In addition, you can use LoRAs to make style / details correctly

MMS text I used for creating Nova Furry XL Illustrious v5.0:

+NF, https://civitai.com/models/503815/nova-furry-xl?modelVersionId=1402403
+SN, https://civitai.com/models/1194136/illustrious-snafu-super-nova-anime-furry-unified
+AL, https://civitai.com/models/1273793/alien-furry-mix-illustrious?modelVersionId=1436977
+YF, https://civitai.com/models/3671/yiffymix?modelVersionId=1447249
+I1, https://huggingface.co/Liberata/illustrious-xl-v1.0/blob/main/Illustrious-XL-v1.0.safetensors

+IE, https://civitai.com/models/1131941?modelVersionId=1423821, %LR
+KF, https://civitai.com/models/1010673?modelVersionId=1132889, %LR

CM NF + AL - SN 0.7 _A
-AL
-SN

CM _A + YF "0,0,0,0,0,0.2,0.2,0.3,0.2,0.2,0,0.3,0.2,0.2,0.1,0,0,0,0,0" _B
-_A
-YF

CM _B +D I1 0.5 0.1 _C
-_B
-I1

LB _C IE:0.6,KF:0.4 _D
-IE
-KF

PR _D _E
-_D

CM _C + _D +S NF "1,0,0,0.3,0.3,0.4,0.4,0,0,0,0,0.3,0.2,0.2,0.2,0.3,0.3,0.3,0.3,0.3" "0,0,0,0,0,0,0,0.2,0.3,0.3,0,0.2,0.2,0.2,0.2,0.4,0.4,0.4,0.4,0.4" NovaFurryILV5

Notebook (edited to have finetune and change DARE alpha into 0.5):

...
%cd /kaggle/working/merge-models

NF = custom_model("https://civitai.com/models/503815/nova-furry-xl?modelVersionId=1402403","NF",mode="checkpoint")
SN = custom_model("https://civitai.com/models/1194136/illustrious-snafu-super-nova-anime-furry-unified","SN",mode="checkpoint")
AL = custom_model("https://civitai.com/models/1273793/alien-furry-mix-illustrious?modelVersionId=1436977","AL",mode="checkpoint")
YF = custom_model("https://civitai.com/models/3671/yiffymix?modelVersionId=1447249","YF",mode="checkpoint")
I1 = custom_model("https://huggingface.co/Liberata/illustrious-xl-v1.0/blob/main/Illustrious-XL-v1.0.safetensors","I1",mode="checkpoint")

IE = custom_model("https://civitai.com/models/1131941?modelVersionId=1423821","IE",mode="lora")
KF = custom_model("https://civitai.com/models/1010673?modelVersionId=1132889","KF",mode="lora")
flush()

!python merge.py "AD" "/kaggle/tmp/models/" "NF.safetensors" "AL.safetensors" --model_2 "SN.safetensors" \
--vae "/kaggle/tmp/vae/VAE.safetensors" \
--alpha 0.7 \
--save_half --prune --save_safetensors --output "TEMP_A"
flush()

TEMP_A=model("TEMP_A",1)

remove_model(AL)
remove_model(SN)
flush()

!python merge.py "WS" "/kaggle/tmp/models/" "TEMP_A.safetensors" "YF.safetensors" \
--vae "/kaggle/tmp/vae/VAE.safetensors" \
--alpha "0,0,0,0,0,0.2,0.2,0.3,0.2,0.2,0,0.3,0.2,0.2,0.1,0,0,0,0,0" \
--fine "1.1,1,1,0.9,1,0,0,0" \
--save_half --prune --save_safetensors --output "TEMP_B"
flush()

TEMP_B=model("TEMP_B",1)

remove_model(TEMP_A)
remove_model(YF)
flush()

!python merge.py "DARE" "/kaggle/tmp/models/" "TEMP_B.safetensors" "I1.safetensors" \
--vae "/kaggle/tmp/vae/VAE.safetensors" \
--alpha 0.5 \
--beta 0.1 \
--save_half --prune --save_safetensors --output "TEMP_C"
flush()

TEMP_C=model("TEMP_C",1)
remove_model(TEMP_B)
remove_model(I1)
flush()

!python lora_bake.py "/kaggle/tmp/models/" "TEMP_C.safetensors" \
"IE.safetensors:0.6,KF.safetensors:0.4" \
--save_safetensors --output "TEMP_D"
flush()

TEMP_D=model("TEMP_D",1)
remove_model(IE)
remove_model(KF)
flush()

!python merge.py "NoIn" "/kaggle/tmp/models/" "TEMP_D.safetensors" None \
--vae "/kaggle/tmp/vae/VAE.safetensors" \
--save_half --prune --save_safetensors --output "TEMP_E"
flush()

TEMP_E=model("TEMP_E",1)

remove_model(TEMP_D)
flush()

!python merge.py "ST" "/kaggle/tmp/models/" "TEMP_C.safetensors" "TEMP_E.safetensors" --model_2 "NF.safetensors" \
--vae "/kaggle/tmp/vae/VAE.safetensors" \
--alpha "1,0,0,0.3,0.3,0.4,0.4,0,0,0,0,0.3,0.2,0.2,0.2,0.3,0.3,0.3,0.3,0.3" \
--beta "0,1,0,0,0,0,0,0.2,0.3,0.3,0,0.2,0.2,0.2,0.2,0.4,0.4,0.4,0.4,0.4" \
--save_half --prune --save_safetensors --output "NovaFurryILV5"
flush()
...

Brief context of the MMS Text:

+NF, https://civitai.com/models/503815/nova-furry-xl?modelVersionId=1402403

Download Checkpoint

+IE, https://civitai.com/models/1131941?modelVersionId=1423821, %LR

Download LoRA

_A

TEMP_A, names that starts with _ becomes TEMP model

-AL

Removes the model: AL

CM _B +D I1 0.5 0.1 _C

Merge Illustrious v1.0 (I1) into TEMP_B using DARE merge
alpha sets to 0.5, likeness value sets to 0.1

LB _C IE:0.6,KF:0.4 _D

Merge LoRA into TEMP_C, IE with alpha 0.6 and KF with alpha 0.4

PR _D _E

Prune TEMP_D and save it as TEMP_E
LoRA Merge results in bigger file so in order to do the further merges and saving, please do this

7. Recommended Resources

Model Merge Scripter - Team C