Experimental
I would not recommend this for anyone but advanced users.
In the train_network.py file find and change the unet and text encoder if desired
unet_weight_dtype = torch.float32
You can see in my training I am running the 32bit optimizer.
I am running the TE in FP8 this may cause problems but it is an experiment.
Save in Float
NOTE:
The training is taking longer then XL and it is on SD 1.5 (768x768 Latent Size)
Update:
I was fully expecting to see the failed training noise images however to my suprise I had good results on epoch 1. I might set this up for an overnight training as it is est 8 hours vs 30 minutes for BF16