Sign In
home
models
images
videos
posts
articles
bounties
tools
challenges
events
shop

Transform Your Drawings into a Luminous Lora with Stable Diffusion

MisterMR's Avatar
MisterMR
Gold Writer Badge
Mar 10, 2024
generation guide
lora guide
Transform Your Drawings into a Luminous Lora with Stable Diffusion

Objective

Stable Diffusion : Create a lora from manual design

Preface

This is an idea to make a Lora from simple drawings. Of course is an idea, I am not a good drawing

Environment

For this test I will use:

Your Drawings

I made a Template to draw images (without coloring them) , the size is square, it is not mandatory but after I will save them 512x512.

(Attached you find the png and the pdf file for this template).

My lora idea (The Hell Lora)

I want to make a lora that represent Hell with horns and the evil tail. Red and fire.

I draw 3 different pieces of paper and took images.

Splitting in images 512x512

Using Kirta I cut and resize each image to 512x512 and I rename them with a progressive number.

Colorize in Stable Diffusion

Set the image in the ControlNet plug-in.

Set Enabled = true and set Control Type=Canny

Set one image and test your prompt.

Prompt

hello red monster,( fire background:1.2) , (smoke atmosphere:1.2) <lora:more_details:1>

Negative

blurry, blurry_image, Lowres, Low_resolution, Low_picture_quality, Low_picture_anime, extra_anatomy, extra_body, extra_navel, extra_face, extra_eyes, extra_chest, extra_nipples, extra_hips, extra_arms, extra_hands, extra_fingers, extra_legs, extra_feet, extra_toe, missing_anatomy, missing_body, missing_navel, missing_face, missing_eyes, missing_chest, missing_nipples, missing_hips, missing_arms, missing_hands, missing_fingers, missing_legs, missing_feet, missing_toe, sketch, sketching, (worst quality: 2), (low quality:2), ((monochrome)), ((grayscale)), inpainting, HDR, bad_prompt_version2, jpeg artifacts, signature, watermark, text, error, missing fingers

hello red monster,( fire background:1.2) , (smoke atmosphere:1.2) <lora:more_details:1>
Negative prompt: blurry, blurry_image, Lowres, Low_resolution, Low_picture_quality, Low_picture_anime, extra_anatomy, extra_body, extra_navel, extra_face, extra_eyes, extra_chest, extra_nipples, extra_hips, extra_arms, extra_hands, extra_fingers, extra_legs, extra_feet, extra_toe, missing_anatomy, missing_body, missing_navel, missing_face, missing_eyes, missing_chest, missing_nipples, missing_hips, missing_arms, missing_hands, missing_fingers, missing_legs, missing_feet, missing_toe, sketch, sketching, (worst quality: 2), (low quality:2), ((monochrome)), ((grayscale)), inpainting, HDR, bad_prompt_version2, jpeg artifacts, signature, watermark, text, error, missing fingers
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 3587511499, Face restoration: GFPGAN, Size: 512x512, Model hash: 80927e9b08, Model: aniverse_v15, ENSD: 31337, Token merging ratio: 0.1, ControlNet 0: "Module: canny, Model: control_v11p_sd15_canny [d14c016b], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Threshold A: 100, Threshold B: 200, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", Lora hashes: "more_details: 3b8aa1d351ef", Downcast alphas_cumprod: True, Eta: 0.5, Pad conds: True, Version: v1.8.0

Here is the output;:

Then select the batch Tab in the ControlNet and set the folder where are the images.

Now run "Generate" to create 9 images for each of your drawings.

Selecting the good images

Now it is time to select the images that I like for my lora. I moved the images that I like to another folder.

Here some of the images that I took as an example:

At the end I have selected 173 files.. some are similar.

Make the Lora structure

I have created the Lora folder strucuture

HelloWorld

  • Img

    • 25_HellWorld_Images

  • Model

  • Log

  • Reg

I copied all the files inside 50_HellWorld_Images folder

Generate the .txt files for each image

We have to make a .txt file for each image where I have to put my KeyWord "HellWorld"

In Stable Diffusion go to the Extras Tab, set the folder with the seleced images (25_HellWorld_Images ) and an output folder (a different empty folder)

Then set the "Caption" and "Deppbooru" or "BLIP" and clic generate.

This will create .txt files and .png files.

Copy the .txt files under 50_HellWorld_Images with the files .

Open each file and add HellWorld at the beginning. I will use notepad++ (https://notepad-plus-plus.org/downloads/) for this .

Of course you have to write the explanation of each image for a better result.

Using Kohya_ss to generate the Lora

Select Lora and Leave the default settings

Set the folders

Here set the folder that we have created the "Output folder" and set the Model folder.

About the parameters I am not an expert.

Attached there is the HellWorld_20240310-163331.json with my settings.

Generate Lora

With this configuration with my graphics card, generation will take about 20 minutes

At the end 23:15 minutes

At the end it generated the file HellWorld.safetensors

Set the lora in Stable Diffusion

Copy the file HellWorld.safetensors in the folder stable-diffusion-webui\models\Lora\HellWorld

Now the lora is ready.

I Have published in https://civitai.com/models/343847?modelVersionId=384903

:

You can check it an let me know.

Result

It was funny to make this lora from zero. Cool to see the result of my easy drawings on new creations.

I hope you enjoy my article.

10

Comments