A Certain Theory for LoRa Transfer

Inspired by the introduction of AnyLora by Lykon and an experiment done by Machi, I decide to further investigate the influence of base model used for training.

Here is the full documentation

https://rentry.org/LyCORIS-experiments#a-certain-theory-on-lora-transfer

On the same entry page I also have other experiments

I focus on anime training here. To quick recapitulate,

• If you want to switch style when switching model, you should use NAI or ACertainty. On the other hand, if you want the trained style to be retained on a family of models, you should use a model that is close to all these models (potentially a merge).

• If you want style of model X when using it, you train on ancestor of X that does not have this style. Especially, if you want to make cosplay images, you should better train on NAI and not train directly on NeverEndingDream or ChilloutMix.

• Don't use SD 1.4/1.5 for anime training in general unless you train something at the scale of WD.

General Advice

• Dataset is the most important. Use regularization set whenever possible. Make sure data are diverse and properly captioned (remember that trigger word learned what is in image but not described in caption).

• Training on higher resolution can enhance background and details but it is not necessarily worth it.

• I really see no difference training on clip 1 or 2. If you see it, please let me know.

I am not able to upload the full resolution image (more than 100mb for each), but you can download the zip and check yourself.

• Images 2-6, made with final checkpoints with weight 1

• Images 7-9, made with intermediate checkpoints

• Images 10-12, made with final checkpoints with weight 0.65