When using a specific layer training method to train a LoRA for Flux,I encountered issues where the model failed to converge and generated images unrelated to my dataset. After a series of troubleshooting and experiments, I discovered some key factors behind the training failure. This chart summarizes my findings, and I hope it helps others avoid the same mistakes.
t5xxl_fp8_e4m3fn_sclaed is not a good choice for training and should be avoided,and the speedup it provides is negligible.
When training on specific layers, you'll need to increase the learning rate and network alpha values.
If you also increase the training resolution, you'll have to raise them even more.
