LoCon is an direct upgrade for LoRa and DoRa is improvement avilable for it.
So one would assume that training LoCon DoRa would be simple matter.
I tried to train style with LoCon and DoRa enables. Results were pretty bad. Does anyone know if this is something that can be fixed?
More dim? More steps? Something else?
Here results from few training attempts at training https://civitai.com/models/1021790
Training settings were same for all except for the settinngs listed below. Training was done with kohya's sd-scripts.
Normal LoRa (dim 32, alpha 16, 15 epochs, batch size 2)
Baseline success (released as v1.8).
LoCon DoRa (dim 24, alpha 12, conv dim 24, conv alpha 12, 12 epochs, batch size 2)
Colors are weird neon bright.
LoCon DoRa (dim 32, alpha 16, conv dim 32, conv alpha 16, epochs 20, batch size 1).
Better than previous. But colors are still weird.
LoCon DoRa (dim 32, alpha 16, conv dim 8, conv alpha 1, 18 epochs, batch size 2)
Colors are still weird and now also too dark.
Example of training settings toml:
bucket_no_upscale = true
bucket_reso_steps = 32
cache_latents = true
cache_latents_to_disk = true
caption_extension = ".txt"
resolution = "1024,1024"
max_token_length = 150
min_bucket_reso = 256
max_bucket_reso = 2048
enable_bucket = true
shuffle_caption = true
debiased_estimation_loss = true
dynamo_backend = "no"
gradient_accumulation_steps = 1
gradient_checkpointing = true
loss_type = "huber"
max_data_loader_n_workers = 0
max_grad_norm = 1
huber_c = 0.1
huber_scale = 1
huber_schedule = "constant"
no_half_vae = true
noise_offset_type = "Multires"
prior_loss_weight = 1
sample_prompts = "D:/output\\sample/prompt.txt"
sample_sampler = "euler_a"
save_model_as = "safetensors"
save_precision = "bf16"
scale_weight_norms = 5
seed = 12345
xformers = true
pretrained_model_name_or_path = "D://Models/Stable-Diffusion/Illustrious-XL-v1.0.safetensors"
network_module = "lycoris.kohya"
network_args = [ "preset=full", "conv_dim=8", "conv_alpha=1", "use_tucker=True", "rank_dropout=0", "bypass_mode=False", "dora_wd=True", "module_dropout=0", "use_scalar=False", "rank_dropout_scale=False", "algo=locon", "train_norm=False",]
network_dim = 32
network_alpha = 16
unet_lr = 1
learning_rate = 1.0
text_encoder_lr = [ 1.0, 1.0,]
lr_scheduler = "cosine"
lr_scheduler_args = []
lr_scheduler_num_cycles = 1
lr_scheduler_power = 1
mixed_precision = "bf16"
fp8_base = true
full_bf16 = true
optimizer_type = "Prodigy"
optimizer_args = [ "decouple=True", "weight_decay=0.01", "d_coef=0.6",
"use_bias_correction=True", "betas=0.9,0.99", "slice_p=11"]
wandb_run_name = "quartet_8"
output_name = "quartet_8"
train_data_dir = "D:/training/quartet"
logging_dir = "D:/training/log"
output_dir = "D:/training/output"
epoch = 15
train_batch_size = 2
max_timestep = 1000
max_train_steps = 5640
save_every_n_epochs = 2