[Negative LoRA] for RouWei - v0.6.1 vpred

WARNING: This LoRA only works with v-pred checkpoints, and is specifically for RouWei v0.6.1 v-pred.

RouWei is an Illustrious based model by Minthybasis: (https://civitai.com/models/950531?modelVersionId=1130140)

Put in your positive prompt:

<lora:rouwei_neg_by_pfc:1>

Put in your negative prompt (optional):

(by pfc:1.0)

Suggested Strength: 0.1 to 1.0

Model Description:

This is an experimental attempt to increase the aesthetic quality of outputs using RouWei. The application of this LoRA can help resolve fine details, fix anatomical details, increase background details, and more.

Version v2 is the first version and there is certainly room for improvement.

Known Limitations:

• Putting "by pfc" in Negative prompt sometimes does f* all

• Increasing strength of LoRA too much can Pony-fy the image, even without trigger words

• Weird artifacts at high CFG scales

Dataset / Training Details:

The dataset is comprised of 360 AI-generated images from CivitAI, primarily from Pony Diffusion XL / Animagine derived models.

The hypothesis is that we can use RouWei's vast artist knowledge to associate the "Pony" style with its own unique token, so we can use it as a negative.

Most, but not all, of the images used for training suffer from any of the following:

• high frequency details

• washed out colors

• subtle bloom / haze

• uncanny or bizarre faces

• boring compositions

• I don't like them

The images were all captioned with "by pfc", because it is a rare token with little effect before training.

More fragments:

• Trained for 60 epochs (out of 100) at batch size 8, gradient accumulation steps 8, on an RTX 4090 (2-ish hours)

• Likely undercooked because I stopped it before reaching 100 epoch

• Only U-net is trained

• Network/Conv Rank 8, Alpha 4 to dampen effect.

• Using the ProdigyWithScheduleFree optimizer (https://github.com/kohya-ss/sd-scripts/issues/1796) because it's the newest and best thing, and v-pred models are hard to train.

• Debiased Estimation Loss is better than Min Gamma SNR

Plans:

• Expand/prune dataset to only the ugliest images

• Try training with DoRA

• Try training with tags (if it doesn't make the result worse)