As of the writing day, hunyuanDit training tool is not very accessible. I write up my custom training script for this experiment.
code repo: https://github.com/lrzjason/T2ITrainer
train_hunyuan_lora_ui.py
Original image:
prompt: cotton doll, A plush toy character with pink hair, a yellow crown adorned with a red gem, and large blue eyes. She is dressed in a white dress with a blue pendant around her neck. Surrounding her are cotton balls, which appear soft and fluffy. The backdrop is a solid peach color, providing contrast to the character's pastel hues. The overall composition gives off a warm and cute vibe, emphasizing the character's innocence and royalty.
hyperparameter:
adamw constant 1e-4 constant repeats 100 epoch 20
The image overfitted at 1600 steps as following and doesn't follow prompt at this checkpoint
prompt: cotton doll, A plush toy character of a policy officer
The 400 steps lora learned the concept and it has pretty good generalization
prompt: cotton doll, A plush toy character of a policy officer
The trainer repo is still wip. So, if things don't work, you might make a issue and let me know. But I don't sure would it be fixed or not.