Sign In

Fine-tune Lora in COMFYUI, no training data

DoKaPirates's Avatar
DoKaPirates
Oct 21, 2023
training guide
comfyui workflowsd1.5lora trainingsdxl lora
Fine-tune Lora in COMFYUI, no training data

Okay guys & gals I need your help. I was fascinated by this video of Harvard showing how bacteria evolve rapidly after each generation, adapting to environments 1000 times more toxic with ease (watch it here - only 2 minutes)

Basically:

  1. Throw bacteria into a toxic environment.

  2. Only the survivors can reproduce.

  3. The offspring then move on to even more toxic environments, with only the fittest able to survive and reproduce.

  4. Repeat.


I want to try something similar for SD:

  1. Begin with 10-20 randomly generated Loras, each creating 10-20 images (totaling 100-200 images).

  2. Pick the top 2 LoRas as parents to pass on their good "genes"

  3. Additionally, pick the bottom 2 LoRas as worst "genes"

  4. These 2 parent Loras reproduce, with some mutation genes, starting the 2nd generation.

  5. Repeat



I call it

LET'S EVOLVE & KICK ASS (LEKA)

Yeah cringe I know, say whatever you want lol.

Initially, I tried training Kohya with random images, but it was too much work and time-consuming, plus it didn't fit into ComfyUI workflow. So, I switched to ComfyUI instead.
It kinda works and is much faster than using Kohya, but still not as fast as I'd like, so I need your help.

QUICK EXAMPLE

Generation 1: Most random Loras show no coily hair unless you enter it in the prompt. At 1st generation, you have to keep creating new random Loras until you got one that shows coily hair.

Generation 5: Most LoRas have 0-1 coily hair images (out of 10-20 images)

Generation 10: Most Loras have 0-2 images

Generation 15: Most Loras have 1-3 images, some even got 4-5 images

You can see that the new Lora doesn't change much of the background, shirt color, poses, etc. of most images.

It takes multiple generations to increase the number of good images (coily hair in this case).
This is just an example, I might have gotten lucky here but some other genes could take many generations to appear and be fully expressed.

DISCLAIMER

  • Yes, it's a clickbait title and I'm sorry. This is just a proof-of-concept with a half-finished Lora still in training. I couldn't wait so I decided to share its workflow first & improve it later.

  • I don't know of any ComfyUI nodes that can mutate a Lora randomly, so I use Lora Merger Node as a workaround. It's slow and keep showing errors, taking 3-5 minutes just to create one Lora (Still, it's much faster & more convenient than using regular trainers like Kohya for this purpose though).

  • I hope some smart folks could figure out how to randomly mutate a Lora, it could cut the time by 90% – no joke

  • We can't fine-tune things from scratch with this method (Well we can, but it would take an insane amount of time and effort).

  • I put some tips at the end

SO WHAT'S THE GOAL?

  1. I want a simple workflow that anyone can use to tweak their Lora and keep it evolving to match our ever-changing taste.

  2. If possible I wish that it could take effect on-the-fly and can be connected to any ComfyUI workflow.

  3. If successful, we might even be able to tweak (or improve) Loras that we can't train currently like:

    • Conceptual elements or new styles that lack training data

    • Lora for SDXL inpainting models

    • Lora for AnimateDiff

    • Well theoretically almost any type of Lora can be improved this way, even the whole model, as long as it can be connected to a workflow allowing it to influence the results

REQUIREMENTS

  1. LORA Merger ComfyUI: https://github.com/laksjdjf/LoRA-Merger-ComfyUI . Seriously, you only need this node to modify the LoRa. Everything else is just like a normal workflow. Use the node you want or use ComfyUI Manager to install any missing nodes.

  2. 60-100 random Loras to create new mutation genes (I already prepared 76 Loras for you)

  3. Optional: install Pythongosssss https://github.com/pythongosssss/ComfyUI-Custom-Scripts there are many useful scripts available such as the feature to follow/jump to the executing node. This helps us understand what's happening and where the flow is stuck.

WORKFLOW FILE

I have shared the workflow json file which includes detailed instructions. I'm not good at explaining things, so if you have any questions please feel free to ask. I'll answer and update this article.

There are only 4 main parts:

  1. BEST GENES: Pick 2 best Loras of each Lora generation (not image generation) and put them there

  2. WORST GENES: (Optional) Pick 2 worst Loras

  3. GENE POOL: For generating random Lora with low weight (0.05-0.15) to act as new mutation genes to be added to the BEST GENES.

  4. KSAMPLER: This is where you create your next generations of LORA. There are 4 options:

    • Merge MUTATION GENE + BEST GENES + WORST GENES

    • Merge MUTATION GENE + BEST GENES

    • Merge MUTATION GENE + BEST GENES 1ST

    • Merge MUTATION GENE + BEST GENES 2ND

  • First, for testing you should select an easy target like shirt color, background or hair.

  • Next, create the 1st generation of Loras (about 10-20 Loras). You can place any Lora in the BEST GENES + WORST GENES (set the maximum/minimum weight to 0 if you prefer not to use those Loras)

  • Generate 10-20 images for each Lora to accurately calculate the frequency of that gene's appearance and evaluate whether each Lora's gene is good or bad (I use RTX 3090 which takes 25-30 seconds to generate 20 pics).

  • Choose the top 2 for Best Genes and the bottom 2 for Worst genes. Now, create the 2nd generation of Loras.

  • Repeat this process.

  • We would need 6-10 generations of Lora to observe an effect. I believe that 40-60 generations would greatly improve the results.

As you can see most of our time is spent merging Loras. If we had a node that could modify Loras randomly & quickly, we could speed up the process (10x faster).

Even better, if that node can modify Checkpoints, LyCoris, TI, etc.

Even better than that, if we can find a way to only save the best Lora (not all 20 Loras of one generation) and replace it with its own parents (the best genes), then we can save so much time and storage.

That's why I decided to stop the test, share it first & improve it later with the help of you guys.



TIPS

  1. Sometimes the gene we need is very rare so we have to change our strategy.

    • For example I have created >1000 image of "woman wearing t-shirt" but never seen a yellow or purple t-shirt

    • Most common color is white (80%) followed by black, blue, pink, and red.

    • So instead of choosing "not yellow" I choose any "ALL WHITE" Lora as the worst gene>> I managed to surpress this gene somehow, leading to the emergence of additional colors like yellow t-shirts

    • This is random though; maybe I just got lucky that time, but that's what makes it so interesting.

  2. ComfyUI has a memory leak bug. If you load too many Loras (>32), you may run out of CPU RAM usage and can only produce 3-4 Loras at a time. Fortunately, Ashleyk has helped me solve this issue. You can check out his code in this thread https://github.com/comfyanonymous/ComfyUI/issues/1462. Also his SD template is one of the most popular templates on Runpod. You should check it out; very quick and convenient.

  3. I have optional mini workflows (on the right) to help with the process: resize files & auto tagging & edit images/loras name

  4. Tip me some Buzz, would you?



36

Comments