Artists, designers, and enthusiasts will find LoRA models highly compelling due to their ability to offer a wide range of opportunities for creative expression. Their ease of use with ComfyUI adds to their appeal. This article explores the exciting possibilities associated with LoRA models in ComfyUI.
What is a LoRA?
Stable Diffusion models are fine-tuned using Low-Rank Adaptation (LoRA), a unique training technique. It offers a solution that is particularly useful in the field of artificial intelligence art production by mainly addressing the issues of balancing the size of model files and training power. With LoRAs, users can make model customizations without putting a heavy strain on local storage resources.Ā
An innovative approach has been developed to address these difficulties. LoRA approaches the adaptation of large-scale pre-trained language models in a novel way. The mechanism operates by imbuing each layer of the transformer architecture with trainable rank decomposition matrices and freezing the pre-trained model weights. By employing this novel methodology, the quantity of trainable parameters is drastically diminished, resulting in enhanced operational effectiveness during both the training and inference stages. LoRA enables the scalable execution of AI model integrations into practical tasks, presenting potential resolutions to current obstacles in the implementation of generative AI models.
The LoRA Theory
A method known as the UNet algorithm is employed in the series of computations that make up one of LoRA's hypotheses. The data is input into the left side, which is then used to do a series of computations; the data is then shown on the right side. The picture's long bars show the computing steps. The effect of the produced images can be altered by modifying the output results, which are achieved by altering the parameter weights inside the pillars. Without affecting any one function, LoRA permits the injection of parameters into any existing layer. The original model is unharmed, it's plug-and-play, and the model is compact, which is advantageous.Ā
Source: LoRA
To understand LoRA's, it's helpful to think of it like a "filter" for a camera. You can see that the Prompt and Seed models are identical in the two pictures down below. On one side, we have the original image without any LoRA effects applied, and on the other, we have the image with the LoRA effect applied in a sketch manner.Ā
See the effects of using LoRA's in your ComfyUI workflow
Where to get LoRAās?
Civitai
Civitai has a thriving community where you can always find fresh and exciting models. To find LoRA models, just simply follow these steps.
Go to civitai.com and search for models.
Download LoRA's from Civitai
Click the Filters > Check LoRA model and SD 1.5 base model and after setting the filters, you may now choose a LoRA.
Download LoRA's from Civitai
Click download either on that area for download. Be sure to remember the base model and trigger words of each LoRA.
Download LoRA's from Civitai
Save your LoRA files (SAFETENSORS File format) because you can use these in your instance of ThinkDiffusion.
HuggingFace
UseĀ this https://huggingface.co/models?other=stable-diffusion&sort=downloads&search=lora when searching for LoRAās.
Make sure that the LoRA keyword is used, and the Stable Diffusion left panel filter is selected.
Download LoRA's from HuggingFace
ChooseĀ a LoRA from thisĀ list of models available. You can determine whatĀ kind of LoRA they are according to the arrowed selection.
Download LoRA's from HuggingFace
For example, in the anime-detailer-xl-lora that was selected, it will redirect you to the LoRA page and you will see a Model Card tab which explains the overview, model details and installation process. The right-hand side shows the trigger words that can be used in the text prompt node.
Download LoRA's from HuggingFace
In the Files and version tab, you can see the list of files and you can download specifically the SAFETENSORS file.
Download LoRA's from HuggingFace
Other sources of LoRA's
Some of the websites like https://prompthero.com/ai-models/lora and https://tensor.art/models may offer a download of LoRA models. Google, Github and Reddit are also helpful in providing LoRA's. But for now, let's hook up to those most visited community websites.
Getting Started with ComfyUI powered by ThinkDiffusion
This is the default setup of ComfyUI with its default nodes already placed.
Load LoRA
Download workflow here: Load LoRA
To use your LoRA with ComfyUI you need this node:
Load LoRA node in ComfyUI
For loading a LoRA, you can utilize the Load LoRA node. By adjusting the LoRA's, one can change the denoising method for latents in the diffusion and CLIP models. Examples of common use-cases include improving the model's generation of specific subjects or actions, or adding the ability to create specific styles. To make even more changes to the model, one can even link several LoRA's together.
You have the option to set the LoRA strength settings to negative numbers. Occasionally, this can lead to fascinating outcomes.
Look for a LoRA to be used. For tutorial purposes, let's use Princess Zelda LoRA from Civitai.com. https://civitai.com/models/4959?modelVersionId=5713
Download LoRA's from Civitai
Check the right panel of Princess Zelda LoRA page before downloading it. Make sure to check it has good reviews, base model, trigger words and LoRA stats.
Download LoRA's from Civitai
After downloading the SAFETENSOR file of Princess Zelda. Launch the ComfyUI machine from ThinkDiffusion. Ā
Launch ComfyUI in ThinkDiffusion
Upload Princess Zelda LoRA into the ComfyUI machine.Ā The upload feature can beĀ seen on the right side of the ComfyUI machine. If hidden just click the My Files icon at the bottom corner of the browser in order to pop-up the upload panel. Then follow the sequence of folders: comfyui > models > Lora >
Uploading your LoRA to ThinkDiffusion
Uploading your LoRA to ThinkDiffusion
Restart the ComfyUI machine so that the uploaded file takes effect. You can click the Restart UI, or you can go to My Machines and stop the current machine and relaunch it ( Step 4).
Restart the ComfyUI in ThinkDiffusion
In the ComfyUI, add theĀ Load LoRA node in the empty workflow or existing workflow by right clicking the canvas > click the Add Node > loaders > Load LoRA. The other way is by double clicking the canvas and search for Load LoRA.
Adding the Load LoRA node in ComfyUI
Adding the Load LoRA node in ComfyUI
Once the Load LoRA node appears, connect it to other existing nodes. Connect it by using the same INPUT and OUTPUT to other nodes by simply dragging the point and the connecting wires will appear.
Using the Load LoRA node in ComfyUI
This is the final setup of the workflow that has a Load LoRA node and is connected to other nodes. Make sure you have selected the correct LoRA_name parameters in Load Lora nodeĀ and trigger words in the Text Prompt node.
Selecting your LoRA in the Load LoRA node in ComfyUI
After the workflow has been setup with the Load LoRA node, click the Queue Prompt and see the output in the Save Image node.
Save image node in ComfyUI
Multiple LoRAās
Download workflow here: Multiple LoRAs
Yes, you heard itĀ right. You can add multiple LoRA's in a single workflow.
Look for new LoRAs in websites such as civitai.com and huggingspace.co and download it and upload the SAFETENSOR files to the My Files of ThinkDiffusion. You can follow the steps here:- Where to Get LoRA?. You can add a desired model to your existing workflow such as hand position LoRA, background LoRA, image effects LoRA, etc.Ā
Go to your workflow and add new LoRA nodes beside the existing LoRA node.
Using multiple LoRA's in ComfyUI
Connect the multiple LoRA nodes in a series connection. Observe the circled selection as it shows the connected input / output colors.
Using multiple LoRA's in ComfyUI
Set the correct LoRA within each node and include the relevant trigger words in the text prompt before clicking the Queue Prompt. Iām using the princess Zelda LoRA, hand pose LoRA and snow effect LoRA. The example below executed the prompt and displayed an output using those 3 LoRA's.
Using multiple LoRA's in ComfyUI
LoRA Stack
Download workflow here: LoRA Stack
By default, there is no stack node in ComfyUI. All you need to do is to install it using a manager. LoRA Stack is better than the multiple Load LoRA node because it is compact, saves space and reduces complexity. Below are theĀ steps on how to get the LoRA Stack nodeĀ and how to use it in the workflow.
Go to Manager
ComfyUI manager
Click Install Custom Nodes
Installing missing custom nodes in ComfyUI
Search for ācomfyrollā in the search box and check the list of available nodes below and click the install.
Searching for the ComfyUI comfyroll node
Donāt forget to restart the ComfyUI machine for the newly installed custom node to take effect. If the node is not yet available in the node list even though the RESTART was already clicked, just click the stop button and relaunch the machine.
Restarting your ComfyUI instance of ThinkDiffusion
Search the LoRA Stack and Apply LoRA Stack node in the list and add it to your workflow beside the nearest appropriate node.
Adding the LoRA stack node in ComfyUI
Adding the LoRA stack node in ComfyUI
The image below is the workflow with LoRA Stack added and connected to the other nodes.
Showing the LoRA stack connected to other nodes
Before clicking the Queue Prompt, be sure that the LoRA in the LoRA Stack is Switched ON and you have selected your desired LoRA. As an example, I used Princess Zelda LoRA, Heart Hands LoRA and Snow Effect LoRA. You can follow this workflow and save the output as many times as you like.
LoRA stack node settings in ComfyUI
LoRA in Efficient Node
Download workflow here: LoRA in Efficient Node
By default, there is no efficient node in ComfyUI. All you need to do is to install it using a manager. Advanced ComfyUI users use efficient node because it helps streamline workflows and reduce total node count. Below are theĀ steps on how to get the Load LoRA within the Efficient Loader and how to use it in the workflow.
Go to Manager
ComfyUI manager
Click Install Custom Nodes
Installing custom nodes in ComfyUI
Search for āefficiencyā in the search box and check the list of available nodes below and click the install.
Adding the Efficiency node in ComfyUI
Donāt forget to restart the ComfyUI machine to take effect on the newly installed custom node. If the node is not yet available in the node list even though the RESTART was already clicked, just click the stop button and relaunch it.
Restarting your ComfyUI instance on ThinkDiffusion
Search the Efficient Loader and KSampler (Efficient) node in the list and add it to the empty workflow.
Efficient Loader node in ComfyUI
KSampler(Efficient) node in ComfyUI
The image below is the empty workflow with Efficient Loader and KSampler (Efficient) added and connected to each other nodes.
ComfyUI workflow with all nodes connected
Another Example and observe its amazing output. Be sure to check the trigger words before running the prompt.
ComfyUI workflow
Training a LoRA (Difficult Level)
For those of you who are ambitious, and want to make your own LoRA, there are 2 guides that you can use to train your own LoRA.
Super simple LoRA training guide
This is a super simple guide for anyone who wants to dive straight in and use the .json settings attached in the guide. You can start training a LoRA within 10 to 20 mins and you donāt have to worry about learning all of the settings, trying to understand what it all means.Ā
https://learn.thinkdiffusion.com/a-super-simple-guide-to-lora-training-on-kohya/Ā
Detailed, in depth, LoRA training guide
This guide is more in depth and attempts to explain what each field does within the Kohya_ss user interface. This is a longer tutorial and requires more time to read and understand the content.
https://learn.thinkdiffusion.com/creating-sdxl-lora-models-on-kohya/Ā
If youāre having issues with installation or slow hardware, you can try any of these workflows on a more powerful GPU in your browser withĀ ThinkDiffusion.
Original article can be found here: https://learn.thinkdiffusion.com/comfyui-loras-the-ultimate-guide/