Lora Training Theory: What Model do I pick to train it on?

Preface:

This is an OPINIONATED guide to what happens to ANIME lora, and will sort of take stab guesses at Realism - because even realism we've used our usuals.

So I wanna do a character or style lora! What model?

First of all for you realism nerds before I go into the anime a pointer is here:

A lot of people are requesting the use of Cyber Realistic, which we've never used. We know you can train on STRONG and steady TRAINED models for Realism - but we've never really tested it.

Even Realistic ones have been trained on ANIMATED base ones. So go with that as you will.

Models that get argued about all the damn time

"DON'T USE NAI ITS STOLEN" - Half the Anime community

"Anylora is a Mix, it's a NICE MODEL but wouldn't that cause issues?"

"Don't use SD 1.5 it breaks hands" - Me, every time I chat with Envy.

"Anything 3 isn't NAI, it's a merge"

"AHHHH NOT ANYTHING 4 THATS A TROLL MODEL"

"Epic V3 is a piece of shit why are you downloading it to use?" Me when I trained the Miqo'te Lora for SCIENCE.

In reality it's an argument about WHAT and where and how and why when in reality the truth is this:

it's a preference.

The Preference vs Science

AnyLora is STABLE, it's great - but it's not AS good at 3D content. We've used AnyLora for many of our FFXIV and Second Life content for personal-ish uses (Our alters, and our Miqo'te).

NAI yes is the leaked Novel AI model, but look - it's a preference, certain models give certain style differences.

So let's dive into the science part - I picked AnyLora, NAI and Epic V3 - because Epic V3 was either gonna work or be an entire meme.

We've trained ONCE on our own models before and it came out.. ok THREE TIMES IN TOTAL by now: Polaris on Malware Dream (which was a fine tune, not a merge - but a fine tune off a merge),

Epic V3 - Why you don't train on a somewhat unstable mix.

Epic v3 is GREAT for generating PICTURES, but not for training a lora on.

This was merely the thing - And i was going to do SD 1.5 to prove a point, but SD 1.5-- i'll get to why that WORKS and doesn't in a moment (but no pictures sorry)

1.0 Strength

FFXIV Male Miqo'te <lora:Miq_Lora_Epicv3:1>

Negative prompt: SuperNeg-Test LessTextPls PoodaFix DrafterNeg Neg-AntiArmor SupaColdShower AntiChild

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 2541007798, Size: 512x768, Model hash: 1e6ae087c4, Model: xvoorMix_v10, Denoising strength: 0.5, Clip skip: 2, ADetailer model: face_yolov8n. pt, ADetailer confidence: 0.3, ADetailer dilate/erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.4, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer version: 23.7.5, Hires upscale: 2, Hires steps: 30, Hires upscaler: 4x-UltraSharp, Lora hashes: "Miq_Lora_Epicv3: a6a612b3076e", Version: v1.3.2, Hashes: {"vae": "513d68f522"}

FFXIV Male Miqo'te, 1boy, male focus, white hair, blue eyes, cavalier hat, hat feather, portrait, red headwear, scar, solo, upper body <lora:Miq_Lora_Epicv3:0.5>

Negative prompt: SuperNeg-Test LessTextPls PoodaFix DrafterNeg Neg-AntiArmor SupaColdShower AntiChild

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1392803015, Size: 768x512, Model hash: 1e6ae087c4, Model: xvoorMix_v10, Denoising strength: 0.5, Clip skip: 2, Hires upscale: 2, Hires steps: 30, Hires upscaler: 4x-UltraSharp, Lora hashes: "Miq_Lora_Epicv3: a6a612b3076e", Version: v1.3.2, Hashes: {"vae": "513d68f522"}

So when DIALED DOWN IT'S SORT OF OK?

but at 1 - You kinda wanna ask what hole it crawled out of.

AnyLora - Stable, but not as clear defined for the type of Lora.

This is NOT CALLING OUT ANYONE. just be clear - Lykon's model is great.

I'm not "PAID" to say so, I'm a weirdo I get it.

It's a bit SOFTER and brighter - it picks up in DIFFERENT details. Yes, this is a merged mix that gets used for training - but in all honesty? It's a strong mix, it's not like ours that fly off into the nether worlds.

It CAN struggle with some concepts, and so with 3D that's not already mixed with other data it's best maybe to leave it up to preference?

FFXIV Male Miqo'te, 1boy, male focus, blue hair, blurry, blurry background, dark skin, dark-skinned male, earrings, facial mark, glowing, jewelry, lips, looking at viewer, neck tattoo, portrait, short hair, slit pupils, smile, solo, tattoo, white eyes <lora:Miq_Lora-NELORATRAIN:1>

Negative prompt: SuperNeg-Test LessTextPls PoodaFix DrafterNeg Neg-AntiArmor SupaColdShower AntiChild

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 2566669320, Size: 512x768, Model hash: 1e6ae087c4, Model: xvoorMix_v10, Denoising strength: 0.5, Clip skip: 2, ADetailer model: face_yolov8n .pt, ADetailer confidence: 0.3, ADetailer dilate/erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.4, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer version: 23.7.5, Hires upscale: 2, Hires steps: 30, Hires upscaler: 4x-UltraSharp, Lora hashes: "Miq_Lora-NELORATRAIN: d619b1760db3", Version: v1.3.2, Hashes: {"vae": "513d68f522"}

As you can see - still strong, and VERY nice.

Negative prompt: SuperNeg-Test LessTextPls PoodaFix DrafterNeg Neg-AntiArmor SupaColdShower AntiChild

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 3996709736, Size: 512x768, Model hash: 1e6ae087c4, Model: xvoorMix_v10, Denoising strength: 0.5, Clip skip: 2, Hires upscale: 2, Hires steps: 30, Hires upscaler: 4x-UltraSharp, Lora hashes: "Miq_Lora-NELORATRAIN: d619b1760db3", Version: v1.3.2, Hashes: {"vae": "513d68f522"}

And yes, this is with OUR negatives because we're brats and we've been trying to develop our own stuff to use with our models :P. FastNegative is great, though sometimes it just flings things in to the same poses -- it happens.

NAI - Finer details. Down to preference? Yes.

Again we're clearly not knocking AnyLora, it's been great for a lot of style choices we've had - but for 3D it might be not as good. But y'know LYKON never promised accuracy on AnyLora for anything BUT Anime! :)

Or at least yknow Anime leaning 3d - rather than FFXIV which is a crossbetween both worlds.

Negative prompt: SuperNeg-Test LessTextPls PoodaFix DrafterNeg Neg-AntiArmor SupaColdShower AntiChild

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1646944508, Size: 512x768, Model hash: 1e6ae087c4, Model: xvoorMix_v10, Denoising strength: 0.5, Clip skip: 2, Hires upscale: 2, Hires steps: 30, Hires upscaler: 4x-UltraSharp, Lora hashes: "Miq_Lora-NAI: db5ab7207fbf", Version: v1.3.2, Hashes: {"vae": "513d68f522"}

Here we have it attempting to do G'raha Tia with random tattoos.

Yea, this isn't NAI or AnyLora's fault, it's our fault because we have A VERY diverse dataset instead of just generic faces.

Negative prompt: SuperNeg-Test LessTextPls PoodaFix DrafterNeg Neg-AntiArmor SupaColdShower AntiChild

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 945625914, Size: 512x768, Model hash: 1e6ae087c4, Model: xvoorMix_v10, Denoising strength: 0.5, Clip skip: 2, Hires upscale: 2, Hires steps: 30, Hires upscaler: 4x-UltraSharp, Lora hashes: "Miq_Lora-NAI: db5ab7207fbf", Version: v1.3.2, Hashes: {"vae": "513d68f522"}

WARNING: this was 0.4 instead of 0.5 but we have others at 0.5 to share quick

Clearly we messed up the tagging so you can't recreate the red mage getup from level 50 - but that's our fault.

FFXIV Male Miqo'te, 1boy, male focus, white hair, blue eyes, cavalier hat, hat feather, portrait, red headwear, scar, solo, upper body <lora:Miq_Lora-NAI:0.5>

Negative prompt: SuperNeg-Test LessTextPls PoodaFix DrafterNeg Neg-AntiArmor SupaColdShower AntiChild

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1392803015, Size: 512x768, Model hash: 1e6ae087c4, Model: xvoorMix_v10, Denoising strength: 0.5, Clip skip: 2, Hires upscale: 2, Hires steps: 30, Hires upscaler: 4x-UltraSharp, Lora hashes: "Miq_Lora-NAI: db5ab7207fbf", Version: v1.3.2, Hashes: {"vae": "513d68f522"}

FFXIV Male Miqo'te, 1boy, black hair, cat boy, dark skin, facial hair, facial mark, hair between eyes, hyur, lips, looking at viewer, male focus, neck tattoo, parted lips, portrait, red eyes, red hair, short hair, simple background, slit pupils, solo <lora:Miq_Lora-NAI:0.5>

Negative prompt: SuperNeg-Test LessTextPls PoodaFix DrafterNeg Neg-AntiArmor SupaColdShower AntiChild

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 2592358892, Size: 512x768, Model hash: 6beaa20ace, Model: brainime_v30, Denoising strength: 0.5, Clip skip: 2, Hires upscale: 2, Hires steps: 30, Hires upscaler: 4x-UltraSharp, Lora hashes: "Miq_Lora-NAI: db5ab7207fbf", Version: v1.3.2, Hashes: {"vae": "513d68f522", "model": "6beaa20ace"}

So what's the science proof?

Epic V3:

Proportions are off, the details are wonky and the original model's clip function just doesn't play ball with a lora training. We weren't planning on releasing this but if you really wanna test it out for your self here's the link:

https://huggingface.co/Duskfallcrew/lora_dump/resolve/main/Miq_Lora_Epicv3.safetensors

AnyLora

Overall the proportions were fine, it missed a lot of clip detail with outfits - keep in mind that most of the ones we shared were not then re-genned on NAI or Anylora, but the same model for the most part. You can see the NAI one's a bit stronger on certain models.

Test for yourself: https://huggingface.co/Duskfallcrew/lora_dump/resolve/main/Miq_Lora-NELORATRAIN.safetensors

NAI

Well the results are here: https://civitai.com/models/105149/male-miqote-ffxiv-lora

Strong detail, so it is indeed a preference. I'd say for this though it depends on your content, and what you're looking for.

CONCLUSION

Stop arguing over who's model is best or un-best. Present it with reasons why something might work better.

Nobody's forcing you to use ANYLORA, it's just default listed in a lot of colab notebooks.

Thank you to HoloStrawberry's DatasetMaker and Colab!

Lora Training Theory: What Model do I pick to train it on?

Preface:

So I wanna do a character or style lora! What model?

Models that get argued about all the damn time

The Preference vs Science

Epic V3 - Why you don't train on a somewhat unstable mix.

AnyLora - Stable, but not as clear defined for the type of Lora.

NAI - Finer details. Down to preference? Yes.

So what's the science proof?

Epic V3:

AnyLora

NAI

CONCLUSION

Comments