One thing that has been giving me software-dev-anxiety getting back into AI image gen is the absolute crazy ability for all this software to be non-deterministic. I remember first finding out that Stable Diffusion could give completely different images with the same seed and just having the usual OCD sinking feeling inside.
It feels like things have improved a little, but still - I regularly find stable-diffusion-webui giving consistent results until I restart it, and then never being able to recreate that one tweakable seed anymore. I know it's complicated software made more complicated by LORAs, samplers, schedulers, versions, controlnets, etc, but it doesn't make it any less pang inducing.
I was basically reminded of this with FLUX LORAs being the new big thing. Comfy-UI has already rewritten the weight/blending algorithm which caused all generations to now be completely different in the current version onwards. Eek. Forge is also showing a similar weighting issue, which means that will change at some point once it's fixed. Forge also has some weird inconsistencies with LORAs between GGUFs and 16/8bit ST which adds to this.
Model training with multiple GPUs is one of the main culprits for me. Because of how the parallel tasks are split, what else the GPU may be doing, etc. even the same seed never produces the same results the same time. It can almost be a dice roll. I could take the some training data and have an incredibly amazing one, and an averagely meh one - same epochs, same source data, same seed.
I know there's no real solution to this and it comes with the territory of large models, quantizing and training, but coming from a software development background (with previous experience with procedurally generated content) and having aspects of OCD, I've always been insanely careful with these kinds of things to make my systems deterministic and results predictable and repeatable.
