Sign In
home
models
images
videos
posts
articles
bounties
tools
challenges
events
shop

mint Candy Mix recipe

mixboy's Avatar
mixboy
Aug 10, 2023
resource guide
model merge
mint Candy Mix recipe

What do you do when you have a model you like, but licensing issues prevent you from using it for your purposes? What if you could create a model similar to that model without the licensing issues? This is not a complete solution, but it might give you another hint.

The model we'll create here is a model that has similar output to Aniverse. Aniverse is a fantastic and very popular model that is halfway between a photorealistic model and an anime model, with a higher percentage of photorealism than other anime models. However, while this model is fine for personal use, it has too many licensing restrictions for commercial use. Here we will attempt to create a model similar to the Aniverse V1.2 output.

Select Models

First, we have to choose suitable models with no licensing issues. It would be nice if the output was somewhat similar. In this case, it is better to check for cosine similarity rather than visual similarity. Here are the cosine similarities of some models between Aniverse.

Aniverse V1.2 vs Anything V3

python similar.py aniverse_V12Pruned.safetensors Anything_v3Fixed-prunedFp16.safetensors -o
seed: 114514
base: aniverse_V12Pruned.safetensors [aa5a7100]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  92.9593% |  92.2664% |  92.3024% |  91.7380% |  93.0020% |  88.7739% |  92.5948% |  92.0022% |  95.2486% |

Anything_v3Fixed-prunedFp16.safetensors [ffd430db] -  92.3209%

Aniverse V1.2 vs candyMix V1

python similar.py aniverse_V12Pruned.safetensors candyMix_V1Fp16.safetensors -o
seed: 114514
base: aniverse_V12Pruned.safetensors [aa5a7100]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  93.5332% |  93.3103% |  90.4914% |  92.6028% |  91.3401% |  92.4647% |  92.2295% |  98.5985% |  98.4806% |

candyMix_V1Fp16.safetensors [1be0d372] -  93.6723%

Aniverse V1.2 vs meinamix V10

python similar.py aniverse_V12Pruned.safetensors meinamix_meinaV10.safetensors -o
seed: 114514

base: aniverse_V12Pruned.safetensors [aa5a7100]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  94.0179% |  93.4003% |  93.6197% |  92.1062% |  96.1811% |  94.3770% |  97.8732% |  94.9970% |  97.3339% |

meinamix_meinaV10.safetensors [c7a66bc6] -  94.8785%

We should also choose a photorealistic model to create a model with a photorealistic character, such as Aniverse. In this case, we chose blessingMix for comparison.

python similar.py aniverse_V12Pruned.safetensors blessingMix_V1Fp16-vae.safetensors -o
seed: 114514

base: aniverse_V12Pruned.safetensors [aa5a7100]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  95.5853% |  94.2190% |  93.5703% |  96.1447% |  93.2883% |  96.5714% |  99.1238% |  98.7316% |  98.4794% |

blessingMix_V1Fp16-vae.safetensors [2e5de471] -  96.1904%

Interestingly, the block-level cosine similarities of Aniverse are very high to blessingMix relatively.

Selected Models

Merge Recipe

To get the block level weights to make an Aniverse like output_blocks, it is necessary to fit the weights against OUT03~OUT11.

  • OUT04~OUT07 : face related blocks

  • OUT03, OUT08~OUT11: other output blocks

Step #0

python fit.py --out 4,5,6,7 aniverse_V12Pruned.safetensors candyMix_V1Fp16.safetensors blessingMix_V1Fp16-vae.safetensors
 * input blocks = []
 * output blocks = [4, 5, 6, 7]
 * selected blocks = ['output_blocks']
 * seed = 114515

- model: aniverse_V12Pruned.safetensors
sd_version = 1.5

- model: v1-5-pruned-emaonly.safetensors

- model: candyMix_V1Fp16.safetensors
 - model = model_a
 - load model_a...

- model: blessingMix_V1Fp16-vae.safetensors
 - model = model_b
 - load model_b...
block = output_blocks, n =4
[0.4 0.1] : 83.9802%
[0.42 0.1 ] : 85.6847%
[0.4   0.105] : 84.3823%
[0.42  0.105] : 85.8861%
[0.43   0.1075] : 86.3445%
[0.45   0.1025] : 87.1297%
[0.475   0.10125] : 88.0431%
[0.485   0.10875] : 88.7721%
[0.5175   0.113125] : 90.0414%
[0.5625   0.106875] : 92.1165%
[0.62875   0.1065625] : 93.7457%
[0.67125   0.1184375] : 95.6369%
[0.769375   0.12703125] : 96.9258%
...

output_blocks.4
97.3638%
tensor([0.7301, 0.1379])
output_blocks.5
97.3126%
tensor([0.6967, 0.1101])
output_blocks.6
98.9259%
tensor([0.7106, 0.1150])
output_blocks.7
98.0466%
tensor([0.6426, 0.1755])

output_blocks.1 and output_blocks 8,9,10,11 run separatly and get the following

output_blocks.8
98.2437%
tensor([0.7081, 0.1329])
output_blocks.9
99.7916%
tensor([0.4467, 0.4172])
output_blocks.10
99.6138%
tensor([0.4326, 0.5071])
output_blocks.11
98.4322%
tensor([1.0590, 0.1450])

and

output_blocks.3
96.8835%
tensor([0.6400, 0.2694])

Step #1

based on these weights, apply weight-sum first (SuperMerger notation)

  • MERGE1 = candyMix_V1Fp16 x (1-alpha) + v1-5-pruned-emaonly x alpha (0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.36,0.2669,0.3033,0.2894,0.3574,0.2919,0.5533,0.5674,0.0)

for the first model candyMix, add v1-5 to reduce the amount of add (OUT04 = 0.2669 = 1 - 0.7301, OUT05 = 0.3033 = 1 - 0.6967, ...)

Step #2

  • MERGE2 = MERGE1 + (blessingMix_V1Fp16-vae - v1-5-pruned-emaonly) x alpha (0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.2694,0.1379,0.1101,0.115,0.1755,0.1329,0.4172,0.5071,0.145)

    • add differences

MERGE2 is the mintCandyMix_V1

Cosine Similarity

output_blocks : Aniverse V1.2 vs mintCandyMix V1

python similar.py aniverse_V12Pruned.safetensors mintCandyMix_V1Fp16.safetensors -o
seed: 114514

base: aniverse_V12Pruned.safetensors [aa5a7100]

|   OUT03   |   OUT04   |   OUT05   |   OUT06   |   OUT07   |   OUT08   |   OUT09   |   OUT10   |   OUT11   |
|  96.6110% |  96.6612% |  97.9282% |  98.6372% |  96.7471% |  99.4361% |  99.0511% |  99.6962% |  98.1257% |

mintCandyMix_V1Fp16.safetensors [f8ab932f] -  98.0993%

Notes

Recommends

0

Comments