LoRa, LoCon, LoHa, LoKr, GloRa

So many different LoRa variations. Are they any good?

Googling for results doesn't yield much good information. So I decided to run some tests myself.

I took the training data used in yozakura quartet LoRa and trained multiple different LoRa types with same training data and more-or-less same training parameters.

~~The trained models are not available currently. Apparently articles can't have large files attached and model page complained about "too many files" when I tried to attach 58 models in there.~~

The trained models are here if you want to test them: https://civitai.com/models/2343120?modelVersionId=2635649

Some steps during training give bad results. Due to low number of saved steps it is possible that I was just unlucky and the bad looking models are just saved at bad spots. Number of comparison images was also relatively low.

Also I am no expert so some of the training settings I chose are based on nothing. Better training settings most likely exist.

TL;DR:

LoKr, GloRa, LoCon are all good. LoKr is maybe best in my tests.

LoHa doesn't feel that good compared to the above. Even LoRa had better results than it.

LoRa did reach results almost as good as the others. But sometimes required more steps to reach same level.

Training TE made LoRa better.

8000 steps was maybe not long enough training. At 2000 steps the results for single character were already good. But for multi character use at least 4000 steps were needed, and it seems like 8000 steps (end of training) was not the peak of improvements. Perhaps even longer training could make things better.

General info

Training data: 754 screenshots from yozakura quartet. Tagging style used was "booru tags of full image contents". Training data is almost identical to what was used here so you can just look at that https://civitai.com/models/1021790/anime-screencap-style-yozakura-quartet

This training data was used because it is known to give pretty good results, and the base model is not good at producing this style and characters.

Each LoRa type was trained for rougly same amount of time (7.5h on RTX 3070). Multiple models were saved mid training at various steps.

See "tomls.zip" download for exact training settings.

Dim 16, alpha 16.

Optimizer used was same for all: prodigyopt.prodigy.Prodigy(decouple=True,weight_decay=0.01,d_coef=0.5,use_bias_correction=True,betas=(0.9, 0.99),slice_p=10,safeguard_warmup=True,growth_rate=1.01).

Batch size 2.

All the memory saving options were enabled in order to make it fit into 8 GB VRAM.

Comparison images

I'll be comparing the models with these prompts using WAI-illustrious-SDXL v1.5. And this is the result without any LoRas.

I did run bunch of test matrices with other prompts and seeds too and they impact my judgement of the models. Dumping them all here would flood the article so you'll have to take my word on them.

yozq, anime screenshot, 1girl, nanami ao, blue hair, medium hair, cat ears, blue eyes:0.6), shorts, t-shirt, balancing on head, object on head, book, smile ,looking at viewer, hand on own chin, blue sky, blue background, standing, upper body

yozq, anime screenshot, 2girls, multiple girls, isone kotoha, green shirt, t-shirt, reen skirt, pleated skirt, brown eyes, nanami ao, black dress , sleeveless dress, cat hat, arm around shoulder, sitting

yozq, anime screenshot, 1girl, kurumaki zakuro, red hair, medium hair, bob cut, yellow eyes, china dress, cleavage cutout, light smile, crossed arms, standing, hips, looking at viewer

Standard LoRa

The normal and most common LoRa.

dim 32, alpha 16.
TE training disabled.
Trained for 10000 steps.
2.47 s/it, 7.4 GB VRAM.
162 MB model size.

nanami ao good after 6000 steps.

kurumaki zakuro looks fine pretty fast. Maybe this was a too simple thing to test?

Isone kotoha and Nanami Ao has lots of bleeding between characters until large number of training steps.

TE training yes/no?

Some people say that training TE makes it better. Some say training TE makes the model blow up on your face.

Same settings as above but this time with TE training enabled.

2.71 s/it, 7.5 GB VRAM.
217 MB model size.

Training speed is noticeably slower.

nanami ao at 1200 steps look nice, Step 8400 looks very good too. But at 4800 steps something awful happens and the output is weirdly corrupted.

kurumaki zakuro about same as before.

Isone kotoha and Nanami Ao are much better with TE. The character bleeding stops at earlier step.

I would say that the result with TE training enabled looks nicer but the training appears to be less stable. Overall I would say TE is a net positive for in this test.

The rest of the LoRa types were all trained with TE training enabled.

LoCon

LoRa with Convolutional layers.

LoCon is often described as "simple upgrade" over normal LoRa.

Extra parameters over LoRa: Convolution dimension 32, convolution alpha 16.

What are good numbers for convolution layers? I don't know, so I just set them to be same as dimension and alpha.

Trained for 8700 steps.
2.99 s/it, 7.5 GB VRAM.
226 MB model size.

Again a bit slower training. Fewer intermediate models were saved.

nanami ao looks fine.

kurumaki zakuro looks fine. I would say it has more of her "attitude" in these images.

Isone kotoha and Nanami Ao again get their bleeding fixed at late steps. Funny how the Isone kotoha and Nanami Ao swap places as steps change.

LoHa

Low-rank Hadamard product

Another upgrade over LoRa. Extra parameters over LoRa: Convolution dimension and convolution alpha.

At first I tried training with both dimensions at 32 and alpha 16. But I ran out of VRAM.

A blog somewhere said that the LoHa dimension is effectively squared. So LoHa with dimension 32 would be "equal" to LoRa with dimension 1024.

So I trained with both dimensions at 16 and alpha 8.

Trained for 6800 steps.
3.95 s/it.
7.3 GB VRAM.
226 MB model size.

This is by far the slowest LoRa type tested.

nanami ao has a problems with details in eyes. Other than that it is "fine" I guess. Nothing really stands out.

kurumaki zakuro looks fine without issues.

kotoha and Nanami Ao look a bit off.

LoHa 2

As I already said earlier, some people say that a LoHa rank is squared. So Maybe the rank in first LoHa test was just too high?

So I trained second LoHa with dim 6 alpha 3.

Note that this was trained for fewer steps to see at what point the training starts to affect things.

The results aren't really better. Also it seems like at around 1600 steps the training is starting to take effect.

LoKr

Low-rank Kronecker product

It is like LoHa, but different. Not much information about it anywhere.

It also has convolution dimension and alpha. But there is also a new "factor" parameter.

"What does the factor do". I don't know. But it seems to make the resulting model smaller. Factor 2 halves the size, facto 4 quarters the size.

Also the convolution dimension may mean something different here.

Also there is "full matrix mode" option. I don't know what it does other than it uses too much VRAM so I can't use it.

Since I couldn't find even a single blog that said anything about the "factor" I trained three LoKr with factors 0, 2 and 4.

Rank 0:
3.24 s/it.
7.5 GB vram.
226 MB model size.

Rank 2:
3.17 s/it.
7 GB Vram.
113 MB model size.

Rank 4:
3,15 s/it.
7.3 GB vram.
57 MB model size.

Differences in training speed and VRAM usage is within margin of error I'd say.

nanami ao look pretty good with factors 0 and 2. With 4 the likeness is reduced noticeably.This one seems to have learned the suit better than others.

kurumaki zakuro looks fine without issues.

kotoha and Nanami Ao image suffers noticeably as factor goes up.

Higher factor does reduce quality but much less than would be expected from the size reduction. Factor 4 also seems to learn slower. But at 8200 steps even that one had decent result. Same quality with smaller size? Black magic.

Perhaps LoKr requires longer training than others? Or maybe it is just more stable during training allowing it to benefit from extra training without frying itself. Stay tuned for part 2.

A bit worrying thing is that sd-webui-forge-classic spits out large list of errors with LoKr. Perhaps it is not well supported?

The errors were like this:

ERROR:ldm_patched.modules.model_patcher:Failed to apply lokr to diffusion_model.output_blocks.8.0.out_layers.3.weight 
view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.`

GloRa

"GLoRA: Generalized LoRA for Parameter-Efficient Fine-tuning". Whatever that may mean. I trained one of them. Its parameter seem to be the same: Convolution and alpha.

3.44 s/it.
228 MB model size.