This is my v1 attempt at a finetune of SDXL's VAE and I also wanted to share the training script. The script itself can be found on github: https://github.com/kukaiN/vae_finetune
I'm doing all this this while on vacation, so apologies for the short description. The finetuning script's readme has info on where the original script came from and the modifications I added to do mixed precision and converting the model keys from diffuser's format to SD's format.
I'm posting v1 of my first attempted vae training, but this one is a failure. I tried finetuning on 5 epoches with anime images (around 60k images from my checkpoint data), When I compare the model weights' difference and the cosine similarity of the underlying weights, I do see that the model was trained, but it seems like training on bf16 and the low lr didn't make the finetuned VAE much different. Although the model hash is different, it's not different enough to produce any noticeable difference.
I plan on training a 2 epoch training version with fp16 training and higher lr to see what happens.