Sign In

[Guide] LoRA Block Weight - a way to finetune LoRAs

226
972
16
Updated: Sep 24, 2023
tutorialextension
Type
Other
Stats
972
Reviews
Published
Mar 21, 2023
Base Model
SD 1.5
Hash
AutoV2
88931884E5
default creator card background decoration
First Birthday Badge
sabin

Reddit version of this guide: https://www.reddit.com/r/StableDiffusion/comments/11izvoj

LoRAs used as example: https://civitai.com/models/7649, https://civitai.com/models/9850

For SDXL model check this guide: https://civitai.com/models/22581?modelVersionId=26964

In a nutshell

Extension name: sd-webui-lora-block-weight

Syntax: <lora:loraname:casyalweight:blockweights>

What is it for?

This extension allows you to connect not the entire LoRA, but only individual blocks. This allows you to use some overtrained models, find a fault in your model, or in some cases combine the best epochs.

For example you can use it to take only initial blocks from LoRA, which have influence on the composition. The last blocks, which mostly determine the color hue. Or the middle blocks. color tone, or the middle blocks, which are responsible for a little bit of everything. This can make it easier to generate things that LoRA wasn't particularly intended, for example:

  1. Lowering the weight of the initial blocks can give you your favorite Anime character with normal proportions.

  2. Lowering the weight of the end blocks allows you to get the same character with eyes half a face, but in a normal color scheme.

  3. Adding end blocks from extraneous LoRAs can enhance stroke, reflections, skin texture, lighten or darken the image

  4. A style that sees everything as homes will slightly reduce its enthusiasm and start drawing characters.

  5. And add all sorts of freaks, artifacts, extra eyes and fingers and stuff. After all, we're going to break the normal workings of the model, by cutting off the pieces you don't like.

Installation

To install, find sd-webui-lora-block-weight in the add-on list and install it.

After restarting the UI, the txt2img and img2img you will see new element: LoRA Block Weight.

Please note: There is currently a conflict with Composable Lora and Additional Networks. Additional Networks right now just broke this extension. Composable Lora can be installed at the same time, only one of them must be Enabled /Activate at a time. Otherwise the effect of the LoRA can be applied twice (if not more), creating a scorched image or a mishmash of colors. This is most likely a Webui problem because prompt scheduling shows similar problems in some conditions.

Off topic, but let me explain. Prompt scheduling is changing a request at a certain step, for example, [cat:dog:0,4] will start drawing the cat, but when 40% of all steps have passed it will remove the cat from the prompt and put a dog in the same place. This can result in an animal that has features of both, as well as and a separately standing badly drawn cat and dog.

Usage

I'll give you a good starting point to start experimenting with block weights:

  1. In the prompt after the name of the LoRA model and weight write another colon and the word XYZ, in the example of the popular model it would be <lora:yaeMikoRealistic_yaemikoMixed:1:XYZ> , or if you check screenshot <lora:HuaqiangLora_futaallColortest:1:XYZ>

  2. After this, make sure that the addon is enabled (Active), expand the XYZ plot of the addon (do not confuse with the X/Y/Z plot in the scripts section) check the XYZ plot option.

  3. Select X Types Original Weights, in the X field enter:

INS,IND,INALL,MIDD,OUTD,OUTS,OUTALL

Preparation is finished, you will see a table like the one attached.

If you like any of the results, replace XYZ in the prompt to the tag, that was at the top of the image, like MIDD:

<lora:HuaqiangLora_futaallColortest:1:MIDD>

If you don't like any of the options, you can try inverting query, all weights will turn into their opposites. To do this instead of XYZ write ZYX and run generation again. There is one small bug: At this point in the article, you need to add one more LoRA with weight 0 and tag XYZ. For example, I took Paimon. I think Paimon was happy that she has weight 0 no matter what. Maybe this will be fixed, maybe it won't. As the author of the add-on explained, this will require a change in the logic of the of the extension.

So example: <lora:HuaqiangLora_futaallColortest:1:ZYX> <lora:paimonGenshinImpact_v10:1:XYZ>

If you like one of the inverted options, You will need to expand below Weights setting list, find in the list the corresponding line, for example MIDD, copy it into notepad/Excel/Word and replace all 1's with any character, all 0's with 1 and the previously specified character to 0, then paste it directly into prompt instead of ZXY. Or you can find ready weights in the comments. Do not forget to remove Paimon from prompt and disable XYZ plot.