(!) These tips are only for LoRA and not for LyCORIS (!)
Tip 1: It's almost never about your training parameters and almost always about the dataset
It's especially easy to tell if you use Prodigy optimizer for training. If your lora didn't learn a character, a concept or a style well in 3000-5000 steps, it's 100% a signal that either your images or captions are bad.
Tip 2: Manual tagging is very important
Before sharing any tips on manual tagging, I want you to understand these simple rules:
Tag what you want your model to learn.
Don't tag what you want to be baked into your LoRA.
If you don't want something baked in and don't want the LoRA to learn it, don't tag it and make sure it isn't present in a significant percentage of your dataset.
Fewer tokens in your captions are better. The exception is style training, where it's important to include a trigger word in every caption and make them as descriptive as possible. For styles, we usually rely on a trigger word to capture the style, and everything else shouldn't be baked in.
The implication of rules 1, 2 and 3 for character LoRA training, you can prevent certain outfit parts from becoming part of the character's default look by tagging them. After training, you can use a negative prompt to remove them. If you don’t tag them, those elements may become strongly attached to the character.
The implication of rule 4 is that the results of auto-captioning should be manually optimized, here are a few examples:
holding, weapon, holding weapon, sword, holding sword, arm up -> holding sword in raised arm (no article because we're optimizing tokens, remember?)
black footwear, boots, black boots, high heel boots -> black high heel boots
The tool I use for manual captioning is Taggui. It truly an incredible tool that allows to do search queries on your dataset like tokens:>150 or tag:"bicycle" AND NOT tag:"bike jersey" . It also allows to add a tag to multiple images in one go. The feature I use often that is not very obvious is that I search for some subset of images, select them and move them to a different folder with an increased number of repeats, so that subset is better learned.
Tip 3: Lower your alpha
The higher your alpha is, the faster your lora will train, with a higher chance of overfitting.
The rule of thumb: if you have 200+ images in your dataset, you will most likely get away with alpha=dim. For smaller datasets lower it to 0.5*dim - 0.75*dim.
Don't be afraid to slow cook your lora with lower alpha and increased number of steps, usually the results are always better this way.
Tip 4: Lower your dim for characters and concepts
You'd be surprised how much dim=8 and dim=16 loras can learn. The reason to lower dim is to prevent lora from learning unwanted details like style and backgrounds.
As a proof, this lora was done with dim=8, it's not perfect in details, but the character is also not your regular waifu. Dim=16 is a good start.
Tip 5: If you have to lower the strength of your lora to 0.6-0.9 to get better results compared to strength=1, it is most probably overcooked
Tip 6: The best parameters for training
Optimizer: Prodigy
simple characters: dim=8 alpha=2-4
complex (or real) characters with many unique details: dim=16 alpha=4-8
simple concepts: dim=8 alpha=4
complex concepts with many variations dim=16-32 alpha=0.5*dim
style having a small dataset dim=32 alpha=16
style having a big (200+) dataset dim=32 alpha=16-32
Tip 7: Learn from others
No, you don't have to ask others. Each lora has a metadata attached to it and this metadata is HUGE, it has all the information you can possibly need. You can see what training parameters were used and how the dataset was captioned.
To read metadata the most handy method is to use this website
Tip 8: Be very picky choosing images for your dataset
Remember, you're going to train a lora to mimic the data from your dataset. You don't want to add images that don't match your expectations 100%. That being said it, you don't want your images to be perfect in every aspect.
For characters and concepts, your goal is to have them represented properly in the dataset, but as long as this goal is achieved it doesn't matter in which styles, in what poses, from which angles they're represented. Quite the opposite, ideally, you want everything except the thing you're training for to be different in each image.
For styles, your goal is to build as much aesthetically pleasant dataset as possible. Don't add low-effort quick draw challenge of an artist to a dataset.
Resolution also plays a huge role in the result you will get, having too many low res images in a dataset might lead to a low quality lora.
Tip 9: Almost all of the advanced settings are just experimental features that were added for testing and research purposes, they won't get you to an advanced level of lora generation, don't even bother to try them out
Tip 10: Upscaling images of a dataset can lead to worse results. Use it with caution.
Upscaling an image will either remove small details or introduce new fake details. Both of these are very undesirable effects when training lora. Upscaling is good if you're training for a concept or if the original doesn't have any details to begin with (cartoon/flat color style).