Is it possible to uncover the chilloutmix recipe using the fit program?
Chilloutmix recipe is shortly described on its own page without further description as follows
"This is Merged "Basilmix" + wonderful realistic models.."
(and published the list of all models.)
On the other hand, there are well-known publicly available models that are very similar to chilloutmix. The followings are the cosine similarities of these models:
chilloutmix and Muse_V1
>python similar.py chilloutmix_NiPrunedFp32Fix.safetensors museV1_v1.safetensors -i -m
seed: 114514
base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]
| IN01 | IN02 | IN04 | IN05 | IN07 | IN08 | MID00 |
| 99.6227% | 99.3470% | 98.5981% | 98.7758% | 97.4793% | 97.6742% | 95.6890% |
museV1_v1.safetensors [f81aba1e] - 98.1695%
>python similar.py chilloutmix_NiPrunedFp32Fix.safetensors museV1_v1.safetensors -o
seed: 114514
base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]
| OUT03 | OUT04 | OUT05 | OUT06 | OUT07 | OUT08 | OUT09 | OUT10 | OUT11 |
| 97.0743% | 96.5173% | 96.8675% | 96.7620% | 98.3015% | 98.5342% | 99.2895% | 98.8423% | 99.3595% |
museV1_v1.safetensors [f81aba1e] - 97.9498%chilloutmix and realmaxV3.4
>python similar.py chilloutmix_NiPrunedFp32Fix.safetensors realMaxV34_v34.safetensors -i -m
seed: 114514
base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]
| IN01 | IN02 | IN04 | IN05 | IN07 | IN08 | MID00 |
| 99.9867% | 99.9938% | 99.9782% | 99.9731% | 99.9560% | 99.9617% | 99.7757% |
realMaxV34_v34.safetensors [4a454569] - 99.9465%
>python similar.py chilloutmix_NiPrunedFp32Fix.safetensors realMaxV34_v34.safetensors -o
seed: 114514
base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]
| OUT03 | OUT04 | OUT05 | OUT06 | OUT07 | OUT08 | OUT09 | OUT10 | OUT11 |
| 99.7540% | 99.7725% | 99.6316% | 99.9358% | 99.9398% | 99.9586% | 99.9749% | 99.9794% | 99.9983% |
realMaxV34_v34.safetensors [4a454569] - 99.8828%But how could they achieve such similarity?
An assumption
How could they achieve such similarity? How did they create such a similar model? Did they ignore copyright and copy without permission, or did they find the recipe in their own way?
Here I'm going to make an assumption: they didn't steal, they found their own recipe, because, although chilloutmix's recipe is not publicly available, they have published the models used to merge it, and it's not such hard to guess the recipe from that information by trial and error many times.
Models
As described in the original document:
Basilmix : nuigurumi/basil_mix @ Hugging Face
the original model was released Jan 2023 with no license restriction.
realistic models:
PoV Skin Texture - r34 Lucid Black @ civitai by twilightBOO
dreamlike license restriction
PoV Skin Texture - Dreamlike r34 @ civitai by twilightBOO
dreamlike license restriction
Procedures
Step #1: Guess merged weights by fit program
Since there are only three models used in the merge, the fit program should work fine, and I tried to fit some blocks as follows:
python fit.py chilloutmix_NiPrunedFp32Fix.safetensors Basil_mix_fixed.safetensors povSkinTextureR34.safetensors povSkinTexture_v2.safetensors --out 3
...(snip)
seed = 114515
message: Optimization terminated successfully.
success: True
status: 0
fun: -0.9999464154243469
x: [ 7.601e-01 1.141e-01 8.267e-02]
nit: 48
nfev: 100
final_simplex: (array([[ 7.601e-01, 1.141e-01, 8.267e-02],
[ 7.602e-01, 1.140e-01, 8.270e-02],
[ 7.601e-01, 1.140e-01, 8.271e-02],
[ 7.602e-01, 1.140e-01, 8.270e-02]]), array([-9.999e-01, -9.999e-01, -9.999e-01, -9.999e-01]))
output_blocks.3
99.9946%
tensor([0.7601, 0.1141, 0.0827])As you can see, the matched similarity is almost 100%!! so I've continued to other blocks and got the following:
output_blocks.4
99.9936%
tensor([0.7608, 0.1181, 0.0776])
output_blocks.5
99.9968%
tensor([0.7667, 0.1002, 0.0948])
output_blocks.6
99.9980%
tensor([0.7542, 0.1173, 0.0812])
output_blocks.7
99.9992%
tensor([0.7619, 0.1075, 0.0869])
output_blocks.8
99.9995%
tensor([0.7603, 0.1111, 0.0858])
output_blocks.9
99.9998%
tensor([0.7660, 0.0969, 0.0994])
output_blocks.10
99.9999%
tensor([0.7474, 0.1343, 0.0690])
output_blocks.11
100.0000%
tensor([0.7050, 0.1405, 0.0883])
and input blocks are:
input_blocks.4
99.9992%
tensor([0.7724, 0.0914, 0.1001])
input_blocks.5
99.9986%
tensor([0.7707, 0.0798, 0.1160])
input_blocks.7
99.9923%
tensor([0.7659, 0.1005, 0.0955])
input_blocks.8
99.9917%
tensor([0.7649, 0.1238, 0.0683])
input_blocks.1
99.9998%
tensor([0.7670, 0.0774, 0.1219])
input_blocks.2
99.9907%
tensor([0.7271, 0.1425, 0.0865])each weight is very similar and the results show almost 100% similarities!
here I made an assumption. Although the weighting ratios are highly deviated, why not have the same ratio of povSkinTextureR34 and povSkinTexture_v2? Two models provide the same effect and there seems no reason to weight each model differently. So I've tried to merge povSkinTextureR34 and povSkinTexture_v2 into pov_merge (50%:50% same weight sum) first. and tried the fit program again.
python fit.py chilloutmix_NiPrunedFp32Fix.safetensors Basil_mix_fixed.safetensors pov_merge.safetensors --out 3,4,5,6,7,10,11
...
output_blocks.3
99.9938%
tensor([0.7637, 0.1974])
output_blocks.4
99.9923%
tensor([0.7684, 0.1949])
output_blocks.5
99.9969%
tensor([0.7671, 0.1951])
output_blocks.6
99.9978%
tensor([0.7616, 0.1981])
output_blocks.7
99.9992%
tensor([0.7677, 0.1932])
output_blocks.10
99.9998%
tensor([0.7672, 0.1959])
output_blocks.11
100.0000%
tensor([0.8806, 0.1205])All ratios are seems similar, so I assume that the original recipe is much easier than expected. I assume the recipe is (basil_mix (1-A) + v1-5 x A)*(1-B) + pov_merge*B
Step #2: finding A and B by trial and error
Based on fitted results, initial A could be 0.05, and B could be 0.2. After some trial and error I got the following:
A: 0.045
B: 0.196
The recipe is
pov_merge = povSkinTextureR34 x 0.5 +povSkinTexture_v2 x 0.5
(basil_mix (1-A) + v1-5 x 0.045) x (1-0.196) + pov_merge x 0.196 = hotChillyMix_V1
v1-5 = v1-5-pruned-emaonly.safetensors
Here are some variations:
A: 0.05, B: 0.2 - hotChillyMix_V1A
A: 0.04, B: 0.2 - hotChillyMix_V1B
Cosine Similarities
python similar.py chilloutmix_NiPrunedFp32Fix.safetensors hotChillyMix_V1Fp16.safetensors -i -m
seed: 114514
base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]
| IN01 | IN02 | IN04 | IN05 | IN07 | IN08 | MID00 |
| 99.9996% | 99.9996% | 99.9958% | 99.9973% | 99.9804% | 99.8364% | 99.9888% |
hotChillyMix_V1Fp16.safetensors [de2f2560] - 99.9711%
python similar.py chilloutmix_NiPrunedFp32Fix.safetensors hotChillyMix_V1Fp16.safetensors -o
seed: 114514
base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]
| OUT03 | OUT04 | OUT05 | OUT06 | OUT07 | OUT08 | OUT09 | OUT10 | OUT11 |
| 99.9925% | 99.9944% | 99.9918% | 99.9949% | 99.9965% | 99.9977% | 99.9994% | 99.9992% | 100.0000% |
hotChillyMix_V1Fp16.safetensors [de2f2560] - 99.9962%Notes
As you can see, the recipe seems very simple
pov_merge ratio seems relatively high and it explains the NSFW property of chilloutmix.
The Base model is the Basil_mix its recipe is not known.
cosine similarity obtained by https://github.com/wkpark/sd-cosine-similarity
SuperMerger also supports cosine similarity analysis
NSFW fix
At that time, one reason for chilloutmix's popularity was its partial support for NSFW (Not Safe for Work) content. As you can see in the recipe, the pov* model was merged a bit, and it complemented the NSFW nicely.
However, compared to recent models, chilloutmix's NSFW support is still lacking. Therefore, I tried to fix the NSFW support.
NSFW fix method
Again, I used the fit program (it will be released on Github soon). The idea is simple, pick a model that supports NSFW well (like as the BlessingMix), and try to fit and merge with the famous sxd model, which is known to support NSFW very well. In other words, using the fit program, try the following. For example, find the x and y weights in the following equation
D = A * x + B * y + C
BlessingMix by mixboy : used for fitting only. not merged in.
sxd v1.0 by @izuek (well-known NSFW model for output_blocks)
D = BlessingMix (good NSFW support destination model)
A = hotChillyMix or base model you have to fix (add difference)
B = selected model to merge to support NSFW (add difference)
C = v1-5-ema-only SD1.5 base model
So, the following fit procedure tries to fit hotChillyMix + sxd_v1.0 to blessingMix.
python fit1.py blessingMix_V1Fp16.safetensors hotChillyMix_V1Fp16.safetensors sxd_10Pruned.ckpt --out 3,4,5,6,7,8,9,10,11
.... (snip)
* seed = 114514
output_blocks.3 : 94.4501%
[1.263191 0.088868]
output_blocks.4 : 94.2208%
[1.339942 0.051585]
output_blocks.5 : 92.7972%
[1.06233 0.127809]
output_blocks.6 : 97.5094%
[1.311994 0.159504]
output_blocks.7 : 98.3428%
[1.113773 0.134744]
output_blocks.8 : 98.8974%
[1.013076 0.126565]
output_blocks.9 : 99.4526%
[1.024839 0.129769]
output_blocks.10 : 99.8654%
[1.015647 0.190674]
output_blocks.11 : 89.8709%
[0.366116 0.229554]
obtained OUT03~OUT11 are
0.088868,0.051585,0.127809,0.159504,0.134744,0.126565,0.129769,0.190674,0.0fitting was done for output blocks, and only the weights of the sxd were used for the merging step.
NSFW fix recipe
based on previous results, the following is obtained carefully by trial and error.
Original weights of OUT03~OUT11 are
0.09,0.05,0.13,0.16,0.14,0.13,0.13,0.19,0.0
fixed initial weights of OUT03~OUT11 are
0.13,0.14,0.15,0.14,0.13,0.17,0.20,0.13,0.0fixed weights after several trial and error
0.34,0.18,0.15,0.14,0.13,0.17,0.20,0.15,0.0
Merging step for NSFW fix
NSFW fixed merge = hotChillyMix_V1Fp16 + (sxd_10Pruned - v1-5-pruned-emaonly) x alpha (0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.12,0.12,0.12,0.34,0.18,0.15,0.14,0.13,0.17,0.2,0.15,0.0)
increase OUT03 to remove strange lumps in the pubic area.
OUT00~OUT02 are chosen as 0.12
Cosine Similarity of NSFW output_blocks
seed: 114514
base: chilloutmix_NiPrunedFp16Fix.safetensors [44b31fce]
| OUT03 | OUT04 | OUT05 | OUT06 | OUT07 | OUT08 | OUT09 | OUT10 | OUT11 |
| 96.2429% | 97.5402% | 98.9703% | 99.0847% | 99.1080% | 99.3181% | 99.7377% | 99.9201% | 100.0000% |
hotChillyMix_V1_NSFW_Fp16.safetensors [5e1733fb] - 98.8802%ChangeLog
2023/08/13 - NSFW fix added. NSFW fix method added.
License
PoV* models have the dreamlike license issue
the original basil_mix was released Jan under the open-rail-m license with no restrictions.
CreativeML-Open Rail-M and Dreamlike Diffusion 1.0 License
This model permits users to:
✔Use the model without crediting the creator
✔Sell images they generate
❌Run on services that generate images for money
✔Share merges using this model
❌Sell this model or merges using this model
❌Have different permissions when sharing merges