Sign In

hot Chilly Mix recipe (aka. chilloutmix clone) and NSFW fix

mixboy's Avatar
mixboy
Aug 18, 2023
resource guide
merge recipe
hot Chilly Mix recipe (aka. chilloutmix clone) and NSFW fix

Is it possible to uncover the chilloutmix recipe using the fit program?

Chilloutmix recipe is shortly described on its own page without further description as follows

"This is Merged "Basilmix" + wonderful realistic models.."

(and published the list of all models.)

On the other hand, there are well-known publicly available models that are very similar to chilloutmix. The followings are the cosine similarities of these models:

chilloutmix and Muse_V1

>python similar.py chilloutmix_NiPrunedFp32Fix.safetensors museV1_v1.safetensors -i -m
seed: 114514

base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]

|    IN01   |    IN02   |    IN04   |    IN05   |    IN07   |    IN08   |   MID00   |
|  99.6227% |  99.3470% |  98.5981% |  98.7758% |  97.4793% |  97.6742% |  95.6890% |

museV1_v1.safetensors [f81aba1e] -  98.1695%
>python similar.py chilloutmix_NiPrunedFp32Fix.safetensors museV1_v1.safetensors -o
seed: 114514

base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  97.0743% |  96.5173% |  96.8675% |  96.7620% |  98.3015% |  98.5342% |  99.2895% |  98.8423% |  99.3595% |

museV1_v1.safetensors [f81aba1e] -  97.9498%

chilloutmix and realmaxV3.4

>python similar.py chilloutmix_NiPrunedFp32Fix.safetensors realMaxV34_v34.safetensors -i -m
seed: 114514

base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]

|    IN01   |    IN02   |    IN04   |    IN05   |    IN07   |    IN08   |   MID00   |
|  99.9867% |  99.9938% |  99.9782% |  99.9731% |  99.9560% |  99.9617% |  99.7757% |

realMaxV34_v34.safetensors [4a454569] -  99.9465%
>python similar.py chilloutmix_NiPrunedFp32Fix.safetensors realMaxV34_v34.safetensors -o
seed: 114514

base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  99.7540% |  99.7725% |  99.6316% |  99.9358% |  99.9398% |  99.9586% |  99.9749% |  99.9794% |  99.9983% |

realMaxV34_v34.safetensors [4a454569] -  99.8828%

But how could they achieve such similarity?

An assumption

How could they achieve such similarity? How did they create such a similar model? Did they ignore copyright and copy without permission, or did they find the recipe in their own way?

Here I'm going to make an assumption: they didn't steal, they found their own recipe, because, although chilloutmix's recipe is not publicly available, they have published the models used to merge it, and it's not such hard to guess the recipe from that information by trial and error many times.

Models

As described in the original document:

Procedures

Step #1: Guess merged weights by fit program

Since there are only three models used in the merge, the fit program should work fine, and I tried to fit some blocks as follows:

python fit.py chilloutmix_NiPrunedFp32Fix.safetensors Basil_mix_fixed.safetensors povSkinTextureR34.safetensors povSkinTexture_v2.safetensors --out 3

...(snip)

seed = 114515
       message: Optimization terminated successfully.
       success: True
        status: 0
           fun: -0.9999464154243469
             x: [ 7.601e-01  1.141e-01  8.267e-02]
           nit: 48
          nfev: 100
 final_simplex: (array([[ 7.601e-01,  1.141e-01,  8.267e-02],
                       [ 7.602e-01,  1.140e-01,  8.270e-02],
                       [ 7.601e-01,  1.140e-01,  8.271e-02],
                       [ 7.602e-01,  1.140e-01,  8.270e-02]]), array([-9.999e-01, -9.999e-01, -9.999e-01, -9.999e-01]))
output_blocks.3
99.9946%
tensor([0.7601, 0.1141, 0.0827])

As you can see, the matched similarity is almost 100%!! so I've continued to other blocks and got the following:

output_blocks.4
99.9936%
tensor([0.7608, 0.1181, 0.0776])
output_blocks.5
99.9968%
tensor([0.7667, 0.1002, 0.0948])
output_blocks.6
99.9980%
tensor([0.7542, 0.1173, 0.0812])
output_blocks.7
99.9992%
tensor([0.7619, 0.1075, 0.0869])

output_blocks.8
99.9995%
tensor([0.7603, 0.1111, 0.0858])
output_blocks.9
99.9998%
tensor([0.7660, 0.0969, 0.0994])
output_blocks.10
99.9999%
tensor([0.7474, 0.1343, 0.0690])
output_blocks.11
100.0000%
tensor([0.7050, 0.1405, 0.0883])

and input blocks are:

input_blocks.4
99.9992%
tensor([0.7724, 0.0914, 0.1001])
input_blocks.5
99.9986%
tensor([0.7707, 0.0798, 0.1160])
input_blocks.7
99.9923%
tensor([0.7659, 0.1005, 0.0955])
input_blocks.8
99.9917%
tensor([0.7649, 0.1238, 0.0683])

input_blocks.1
99.9998%
tensor([0.7670, 0.0774, 0.1219])
input_blocks.2
99.9907%
tensor([0.7271, 0.1425, 0.0865])

each weight is very similar and the results show almost 100% similarities!

here I made an assumption. Although the weighting ratios are highly deviated, why not have the same ratio of povSkinTextureR34 and povSkinTexture_v2? Two models provide the same effect and there seems no reason to weight each model differently. So I've tried to merge povSkinTextureR34 and povSkinTexture_v2 into pov_merge (50%:50% same weight sum) first. and tried the fit program again.

python fit.py chilloutmix_NiPrunedFp32Fix.safetensors Basil_mix_fixed.safetensors pov_merge.safetensors --out 3,4,5,6,7,10,11

...

output_blocks.3
99.9938%
tensor([0.7637, 0.1974])
output_blocks.4
99.9923%
tensor([0.7684, 0.1949])
output_blocks.5
99.9969%
tensor([0.7671, 0.1951])
output_blocks.6
99.9978%
tensor([0.7616, 0.1981])
output_blocks.7
99.9992%
tensor([0.7677, 0.1932])
output_blocks.10
99.9998%
tensor([0.7672, 0.1959])
output_blocks.11
100.0000%
tensor([0.8806, 0.1205])

All ratios are seems similar, so I assume that the original recipe is much easier than expected. I assume the recipe is (basil_mix (1-A) + v1-5 x A)*(1-B) + pov_merge*B

Step #2: finding A and B by trial and error

Based on fitted results, initial A could be 0.05, and B could be 0.2. After some trial and error I got the following:

  • A: 0.045

  • B: 0.196

The recipe is

  • pov_merge = povSkinTextureR34 x 0.5 +povSkinTexture_v2 x 0.5

  • (basil_mix (1-A) + v1-5 x 0.045) x (1-0.196) + pov_merge x 0.196 = hotChillyMix_V1

    • v1-5 = v1-5-pruned-emaonly.safetensors

Here are some variations:

  • A: 0.05, B: 0.2 - hotChillyMix_V1A

  • A: 0.04, B: 0.2 - hotChillyMix_V1B

Cosine Similarities

python similar.py chilloutmix_NiPrunedFp32Fix.safetensors  hotChillyMix_V1Fp16.safetensors -i -m
seed: 114514

base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]

|    IN01   |    IN02   |    IN04   |    IN05   |    IN07   |    IN08   |   MID00   |
|  99.9996% |  99.9996% |  99.9958% |  99.9973% |  99.9804% |  99.8364% |  99.9888% |

hotChillyMix_V1Fp16.safetensors [de2f2560] -  99.9711%

python similar.py chilloutmix_NiPrunedFp32Fix.safetensors  hotChillyMix_V1Fp16.safetensors -o
seed: 114514

base: chilloutmix_NiPrunedFp32Fix.safetensors [95f8d0a7]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  99.9925% |  99.9944% |  99.9918% |  99.9949% |  99.9965% |  99.9977% |  99.9994% |  99.9992% | 100.0000% |

hotChillyMix_V1Fp16.safetensors [de2f2560] -  99.9962%

Notes

As you can see, the recipe seems very simple

NSFW fix

At that time, one reason for chilloutmix's popularity was its partial support for NSFW (Not Safe for Work) content. As you can see in the recipe, the pov* model was merged a bit, and it complemented the NSFW nicely.

However, compared to recent models, chilloutmix's NSFW support is still lacking. Therefore, I tried to fix the NSFW support.

NSFW fix method

Again, I used the fit program (it will be released on Github soon). The idea is simple, pick a model that supports NSFW well (like as the BlessingMix), and try to fit and merge with the famous sxd model, which is known to support NSFW very well. In other words, using the fit program, try the following. For example, find the x and y weights in the following equation

D = A * x + B * y + C

  • BlessingMix by mixboy : used for fitting only. not merged in.

  • sxd v1.0 by @izuek (well-known NSFW model for output_blocks)

    • D = BlessingMix (good NSFW support destination model)

    • A = hotChillyMix or base model you have to fix (add difference)

    • B = selected model to merge to support NSFW (add difference)

    • C = v1-5-ema-only SD1.5 base model

So, the following fit procedure tries to fit hotChillyMix + sxd_v1.0 to blessingMix.

python fit1.py blessingMix_V1Fp16.safetensors hotChillyMix_V1Fp16.safetensors sxd_10Pruned.ckpt --out 3,4,5,6,7,8,9,10,11
.... (snip)

 * seed = 114514
output_blocks.3 : 94.4501%
[1.263191 0.088868]
output_blocks.4 : 94.2208%
[1.339942 0.051585]
output_blocks.5 : 92.7972%
[1.06233  0.127809]
output_blocks.6 : 97.5094%
[1.311994 0.159504]
output_blocks.7 : 98.3428%
[1.113773 0.134744]
output_blocks.8 : 98.8974%
[1.013076 0.126565]
output_blocks.9 : 99.4526%
[1.024839 0.129769]
output_blocks.10 : 99.8654%
[1.015647 0.190674]
output_blocks.11 : 89.8709%
[0.366116 0.229554]

obtained OUT03~OUT11 are
0.088868,0.051585,0.127809,0.159504,0.134744,0.126565,0.129769,0.190674,0.0

fitting was done for output blocks, and only the weights of the sxd were used for the merging step.

NSFW fix recipe

based on previous results, the following is obtained carefully by trial and error.

  • Original weights of OUT03~OUT11 are 0.09,0.05,0.13,0.16,0.14,0.13,0.13,0.19,0.0

  • fixed initial weights of OUT03~OUT11 are 0.13,0.14,0.15,0.14,0.13,0.17,0.20,0.13,0.0

  • fixed weights after several trial and error 0.34,0.18,0.15,0.14,0.13,0.17,0.20,0.15,0.0

Merging step for NSFW fix

NSFW fixed merge = hotChillyMix_V1Fp16 + (sxd_10Pruned - v1-5-pruned-emaonly) x alpha (0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.12,0.12,0.12,0.34,0.18,0.15,0.14,0.13,0.17,0.2,0.15,0.0)

  • increase OUT03 to remove strange lumps in the pubic area.

  • OUT00~OUT02 are chosen as 0.12

Cosine Similarity of NSFW output_blocks

seed: 114514

base: chilloutmix_NiPrunedFp16Fix.safetensors [44b31fce]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  96.2429% |  97.5402% |  98.9703% |  99.0847% |  99.1080% |  99.3181% |  99.7377% |  99.9201% | 100.0000% |

hotChillyMix_V1_NSFW_Fp16.safetensors [5e1733fb] -  98.8802%

ChangeLog

  • 2023/08/13 - NSFW fix added. NSFW fix method added.

License

PoV* models have the dreamlike license issue

the original basil_mix was released Jan under the open-rail-m license with no restrictions.

CreativeML-Open Rail-M and Dreamlike Diffusion 1.0 License

This model permits users to:

✔Use the model without crediting the creator

✔Sell images they generate

❌Run on services that generate images for money

✔Share merges using this model

❌Sell this model or merges using this model

❌Have different permissions when sharing merges

16

Comments