So in the past few weeks I have been dedicating long hours into finding optimal approaches to preserve as much of the ref latent and basically force the model to do two things; preserve the exact features and be flexible.. and it has been such a pain but I think I stumbled accidentally at many interesting features of this model and it’s architecture.. as I tinkered with every possible corner you can tinker with from conds to attn layers to all q,k,v … double and single blocks and more.. overall all I found some valuable information for people who would like to train loras and knowing what to actually target.. and I was wrong while back by publishing a map of where the character lives.. anyways here we go:
Double blocks 0-1 is just base early on where the model is just doing its thing, poses and such are beginning to form here.
Double 2-3 is where the model recognizes the colors of outfit but no outfit / character yet.
Double 4-5 is where the model locks the outfit/ body proportions but not the character’s facial features.
6-7 is where the model locks the character/outfit/features.
Singles 0-23 all just model’s style and textures no actual physical changes nor proportions or features .
And finally yes I need a break from this model.. 😂
I ran a batch for fun with these layers and the results are clean without destroying the model's knowledge or composition, it made it feel like my character was pre-trained with original flux2klein :D
network:
type: "lora"
linear: 32
linear_alpha: 32
conv: 16
conv_alpha: 16
lokr_full_rank: true
lokr_factor: -1
network_kwargs:
ignore_if_contains: []
only_if_contains:
- "double_blocks.6"
- "double_blocks.7"
- "single_blocks.0"
- "single_blocks.1"
- "single_blocks.2"
- "single_blocks.3"
- "single_blocks.4"
- "single_blocks.5"
- "single_blocks.6"
- "single_blocks.7"
- "single_blocks.8"
- "single_blocks.9"
- "single_blocks.10"Config file for who want's to test it out : https://pastebin.com/qAP6AJia