This is an experiment lora only train with 1 image.
adamw constant 1e-4
trained with my custom training script https://github.com/lrzjason/T2ITrainer
The lora is a dora with diffusers format.
You could load the lora with the custom hunyuanDit pipeline in T2ITrainer.
It added the SD3LoraLoaderMixin to the pipeline.
Code:
lora_state_dict = HunyuanDiTPipeline.lora_state_dict(lora_path) HunyuanDiTPipeline.load_lora_into_transformer(lora_state_dict,transformer_)