Ref: My github article, this, this, and then this. The "Karmix" refers to this, the author decided to release here.
This is a progress report more than a tech report. Meanwhile, I admire that this time I'm relying on my art sense instead of my AI / ML knowledge you won't see equations here.
Warning: You won't understand the entire article unless you have read my previous articles, or tried my models, or just experience with my merging / training techniques. You may able to proof that I'm not relying on LLM (vibe research) but these are all original contents and theories. Meanwhile it is more on general AI ML rules instead of specific papers in arxiv. You may use LLM to "decompress" my contents, and hopefully gather some concepts in between.
Changelog:
v1: Initial content.
1. Problem: Turning EPS model to VPred model is hard.
Applied to AstolfoCarmix-VPredXL (AK-NIL1.5 Based).
The original paper was just casually discuss the math equations, then suddenly it became the SD2.0 which was DOA, and the trainer bugs made almost all the downstream finetunes DOA also, until the "zt-SNR" patch (math patch) and NoobAI use other working trainers.
Despite there are a few SDXL non NoobAI Vpred models, which may requires customized code to run, it is not intuitive to "convert" (or finetune to predict other objective variables) because there may be dataset requirement.
After a dozen of trials, especially revisiting "eps-eps" / "vpred-vpred" for validating bug-free codes, here is some key conditions I have found.
Found conditions to turn EPS model into VPred model
Warning: Not vigirously proven!
Subset of the EPS dataset, at least aligned for a large extent. This includes guessing what images has been dropped, and the caption text behind the image. One of the easiest route is the 1girl AI Slop from that model. My friend just used 1k slops to get it converted, leaving me failed with 6k images, even 12.4M images are far worse. Sure, model collaspe may happen, but it really needs a good bias for the new math variable. Otherwise, you are actually pretraining the entire model in math level.
(Not verified by me) Train only part of the model as OUT08 and OUT02. This more on believing on MBW magic. This is heard from NoobAI Discord Server.
Legit training progress with evaluation. Image can break, but cannot turn back to abstract art. Loss curve has no correlation to the learning progress. It is fine if you see the distortion is limited into color fragments. You should stop training once it lost shapes, which can be tested in early stage.
Merge with base model. If you cannot obtain the dataset for the base model (e.g. original dataset from Pony / NoobAI), just don't be purist on model training. You need to manual search for the optimal hyperparameter.
VPred Trainer and WebUI working as intended. Notice that ComfyUI has different behaviour over A1111 and ReForge, which ComfyUI applied EPS at some moment. I prefer test with A1111 since it relies on original codes from StabilityAI (hence the crafted yaml file).
If you are going big and making another SDXL VPred finetune, I have a bad news for you. Enjoy the glitched images. Therefore I quit the VPred conversion until I have my next 1EP done. After a few tries, "task failed successfully" with a solid NaN.
Coming soon: AstolfoXL 2EP, "Evo" Merges, and "ACEvo"
779k steps in total.
