PonyXL-GEM-T2_2

An experimental LoCon trained on outputs from my MIX-GEM-T2_2 model (and a few other MIX-GEM outputs to make up the gap). I spent a lot of time finetuning that model to my ideal aesthetic and I'd rather try to retrieve the style directly from the model than try to remix on a new SDXL base from scratch. Outputs are not very clean and this LoCon has a lot of issues. I will likely have to regenerate the dataset a couple of times with cleaner outputs. Likely there will be a lot of versions of hits LoCon, this will be an iterative process with a lot of rebakes.

Insights gleaned from prototyping:

• Prodigy is worse than AdamW8bit at training style LoCons on PonyXL, even at a higher learning rate it retains a lot less than AdamW8bit. But it also destroys the base model's posing a lot faster, whereas the prodigy tends to keep a lot better with the original posing.

• LoCons are better at training for styles than LoRAs.

• Style retention comes hand in hand with magnifying small mistakes. This isn't a huge issue with ordinary style training, but is extremely problematic when training on SD1.5 outputs because of the way that unnecessary noise gets diffused into random elements which don't really makes aesthetic sense. Case has to be put into selecting only clean outputs.

Things to try in the future:

• White background regularization images

• Hiding hands as much as possible

• Using copyright characters as part of the dataset

After testing, for some reason this LoCon works poorly on autismMixSDXL which washes out a lot of the details, but works extremely well on 4th tail.