Lora training guide - RZ Passage ( 2.1 / 1.5 )

Welcome to this RZ Passage training guide! In this tutorial, I will guide you through the essentials of a standart Lora training. Whether you're a beginner or looking to enhance your skills, this tutorial is designed to provide you with comprehensive knowledge and practical experience. Let's get started!

🧰 Required tools

• Khoya_SS (bmaltais fork)

• Automatic1111

• CLIP Interrogator extension

• A checkpoint (I trained / generated on Realistic Vision V2.0)

🖼️ Dataset overview

The dataset consists of a mixture of fantasy portals and conceptual art. The goal of the training is to generate a scene with a random shape at the center, allowing the checkpoint and the Lora model to be as creative as possible. Care has been taken to include various shapes of portals in order to avoid the training from learning anything specific other than scene composition.

To ensure a versatile Lora model that can think outside the box, it is important to incorporate a variety of scenes, including different environments, portal shapes, themes, and color schemes. This is my vision for what I want this Lora model to achieve. If you prefer minimalistic portals, for example, you can adapt your dataset by including non-realistic pieces, etc.

It's worth noting that recurring elements in the graphics are something that every type of training pays attention to, whether the captions are perfect or not. If the dataset consists of 4 circles and one triangle, the generated outputs are likely to feature more circles than triangles.

✒️ Rewrite the perception (Captionning)

Everything that we do not explicitly caption will be absorbed by the model through our unique identifier.

We need to bypass all terms such as "surrealism, portal, conceptual art" and all the shapes and positions that the CLIP model could discover.

For Lora, the captioning will focus on what we don't want the training to learn and what we want to be interchangeable or replaceable. Since we only want a specific shape in the scene, we will explore different possibilities. 

I will primarily remove artists identified by the interrogator, as their tokens may introduce critical biases towards the shapes we intend to generate after the training.

Let’s help ourselves by loading Interrogator Extension then

1. Prompt tab

2. “Best” mode 

3. Select Vit-L / openai

SD 2.1 Note : Select Vit-H / laion2b, if you have a 24gb card use Best, otherwise "classic".

1. a futuristic city at night with neon lights, cd cover artwork, pyramid hoodvisor, by Wolfgang Zelmer, retrowave color palette, dark ruins landscape, artwork empty daylight, future funk, illuminated features, dark setting

Final caption :

rz88p4ss4ge, a futuristic city at night with neon lights, artwork, retrowave color palette, dark landscape, artwork, illuminated features, dark setting

2. a dirt road in the middle of a forest, official artwork, fantasy game spell symbol, large path, archways made of lush greenery, infinite corridor, gwent, wanderer, green wall, pathways, the endless, gate, forests

Final caption :

rz88p4ss4ge, a dirt road in the middle of a forest, artwork, mist, lush greenery, wanderer

3. a desert scene with a hole in the ground, best of behance, looking through a portal, mind-bending digital art, cactus, the entrance of valhalla, doves flying into the portal, by Carles Delclaux Is, affinity photo, desolated wasteland, gateway to another dimension, otherwordly concept, gate

Final caption :

rz88p4ss4ge, a desert scene, best of behance, digital art, cactus, affinity photo, desolated wasteland

4. a futuristic entrance in the middle of a forest, dribbble 8k, the gates of argonath, magenta and crimson and cyan, panoramic anamorphic, cgi art, by Nathan Wyburn, forbidden - knowledge, beautiful singularities, highly photographic render, beautiful retro art, —n 9, metaverse

Final caption :

rz88p4ss4ge, a futuristic forest, dribbble 8k, magenta and crimson and cyan, cgi art, highly photographic render, beautiful retro art

5. a painting of an arch in the desert, baroque vaporwave statue, hyper realistic digital art, (synthwave), kamelot album cover, white stone temple ruins, 8k archival print, by Wolfgang Zelmer, throne of olympus, by Eric Dinyer, series of dreams, neosurrealism

Final caption :

rz88p4ss4ge, a painting of a desert, baroque vaporwave, hyper realistic digital art, white stone temple ruins, 8k archival print

6. an arch in the middle of a desert with mountains in the background, beeple and mike winkelmann, char aznable album art, very very roberto ferri, ( ( ( in a dark, keyhole, album art for a trance dj, trending on logostation, ( ( ( synthwave ) ) ), 7 0 s art deco, doors

Final caption :

rz88p4ss4ge, a desert with mountains in the background, 7 0 s art deco

7. a picture of an open doorway with mountains in the background, style hybrid mix of beeple, fantasy forest landscape at day, artwork about a road to freedom, photorealistic music album cover, images on the sales website, stone pathways, keyhole, album cover, roads

Final caption :

rz88p4ss4ge, a picture with mountains in the background, fantasy forest landscape at day, artwork, photorealistic

8. an arch in the middle of a desert with a sky background, stunning fantasy 3 d render, roman pillars, ( ( mads berg ) ), palace, as an offering to zeus, 1:1 album artwork, by Abraham Begeyn, opera, ash thorp khyzyl saleem, ozymandias

Final caption :

rz88p4ss4ge, a desert with a sky background, stunning fantasy 3 d render, roman pillars, palace

9. a snowy scene with an arch leading to a snowy mountain, 3 d bold, containing a hidden portal, artforum, promotional images, technology meets fantasy, open door, fantasy land, visual key, the window, gatekeeper

Final caption :

rz88p4ss4ge, a snowy scene leading to a snowy mountain, 3 d bold, artforum, promotional images

10. a picture of a portal in the middle of a forest, trend on behance 3d art, in style of james paick, the concept of infinity, inspired by Gerbrand van den Eeckhout, trending on artstaion, wreath of ferns, enchanted dreams. instagram, gate, by Tom Bonson, 3d cg

Final caption :

rz88p4ss4ge, a picture of a forest, trend on behance 3d art, trending on artstation, wreath of ferns, instagram, 3d cg

11. a digital painting of a futuristic landscape with mountains and mountains in the background, soft lighting album cover, doors that are cosmic portals, projection mapping, cube shaped, bass music, crystal desert, gate, polygon, folded, twitter pfp, door, anvil

💡 I'm excluding the mention of "cube-shaped" here because I'm considering the main "entrance" to not be cube-shaped, but its elements are.

Final caption :

rz88p4ss4ge, a digital painting of a landscape with mountains in the background, soft lighting, cube shaped, crystal desert

12. two tall buildings in the middle of a desert, official artwork, the energy is released, war of colors, crown of white lasers, the entrance of valhalla, 834779519, mirror background, tripmachines, large gate, solid dark background, vhs

💡 A bit hard to caption considering the concept I want to create but well, since i’m not specifying an entrance, a door and a path I consider that the training will pretty much understand what I want

Final caption :

rz88p4ss4ge, two tall monoliths in the middle of a desert, official artwork, the energy is released, war of colors, crown of white lasers, solid dark background

13. a large circle of lightning in the middle of a desert, tooth wu : : quixel megascans, album cover design, portal opening, apocalypse hurricane storm, in the movie arrival, official poster artwork, void vortex, doomsday, otherworldly visuals, dying earth

💡 I could’ve let “void vortex, doomsday” without harms I guess, i just think that redefining the “lightning” and “hurricane storm” is clearly enough

To be honest this one was confusing but I really wanted to have it in the dataset, but i’m pretty sure that it will mess the model, and I like it !

Final caption :

rz88p4ss4ge, a large lightning in the desert, apocalypse hurricane storm, artwork, dying earth

14. a painting of a giant rock formation in the middle of a body of water, anton fadeev and dan mumford, dark red and black color palette, art piece frame, symmetry features, on the coast, magenta and blue, 1801446143, stylized dynamic folds, 2019, cinematic ， - h 7 6 8

Final caption :

rz88p4ss4ge, a painting of a rock formation in a body of water, dark red and black color palette, art piece, on the coast, magenta and blue, 2019, cinematic

15. a painting of a mountain landscape with a pyramid in the middle, iron arc gate door texture, ash thorp, 3 d stylize scene, doors that are cosmic portals, dark enclosed, psytrance, realistic windows, award-winning render, surrealisme aesthetic, by Robert C. Barnfield, featured on artsation

Final caption :

rz88p4ss4ge, a painting of a mountain landscape, 3 d stylize scene, award-winning render

16. a futuristic city in the middle of a desert, soft lighting album cover, tall flowers, from league of legends, edm, beings of astonishing structure, empty space background, album art for a trance dj, utopian, futuristic tower, highly aesthetic, heavenlydaemonic, ruin

Final caption :

rz88p4ss4ge, a futuristic city in a desert, soft lighting, tall flowers, beings of astonishing structure, empty space background

Note: Do not overthink too much over captions just follow this simple logic, since we want a concept, caption the basics of the scene and the main movement / style ( I want this concept to be applicable for realism, paintings etc, so I prune it in my caption).

If we wanted a specific graphic style we would describe more the details of the scene, and no mention over the movement / style / colors at all.

Khoya SS

Basic settings

Source model loading :

Start by opening the Dreambooth LoRA tab and selecting your pretrained model. Alternatively, you can use the model quick pick to train the LoRA on a "raw" pretrained model, such as Runway or Stability-Ai.

SD 2.1 Note : if you train a Lora with a 2.1 based checkpoint check v2 and v_parameterization

Folder settings : 

In image folder, you must select or enter the root folder, where it contains your concept folder

The path specified can’t be “dataset\10_rz88p4ss4ge”.

The "10" represents the number of repetitions per image and works in conjunction with the number of epochs you choose for the final step count. For example, with 10 repetitions and 12 epochs, there will be a total of 120 repetitions by the 12th epoch.

📁 dataset 

   📁 10_rz88p4ss4ge

      🖼️ 01.png

      📄 01.txt

Training settings

In "Advanced", I have chosen the noise offset to be 0.03, which I consider a safe value for this type of concept. It introduces different noise at each iteration of a dataset picture, which helps avoid overfitting issues.

Depending on the average contrast of your dataset, a higher noise offset value may make your Lora model appear brighter or darker.

For managing VRAM, if you are running out of memory with these settings, follow these priorities:

1. Change the optimizer from AdamW to AdamW8Bit.

2. Enable gradient checkpointing in the advanced settings. (only if AdamW8Bit is not enough)

🔍 Training result

Lora strength Grid on the 12th epoch

Steps: 12, Sampler: DPM++ SDE Karras, CFG scale: 4, Seed: 500, Size: 512x512, Model hash: e6415c4892, Model: realisticVisionV20_v20NoVAE, AddNet Enabled: True, AddNet Module 1: LoRA, AddNet Model 1: rz-passage-15-guide(d44cd853d464), AddNet Weight A 1: 0.2, AddNet Weight B 1: 0.2, Script: X/Y/Z plot, X Type: Seed, X Values: "500, 501, 502, 503", Fixed X Values: "500, 501, 502, 503", Y Type: AddNet Weight 1, Y Values: "0.2,0.3,0.4,0.5,0.6,0.7"

rz88p4ss4ge, A landscape of a desert

We can clearly observe the morphing progression in the desert area, where the 0.6 strength images closely fit what I would expect regarding the dataset.

rz88p4ss4ge, a <x> ( prompt S/R ) // Lora Strength 0.55

A wonderful Passage grid ! With minimal prompting to reflect the "core" of the lora.

Now you know how to train a conceptual model, which allows you to modify the original checkpoint seed with different shapes.

Disclaimer: I'm not claiming that my process is the "best way," as it may vary depending on the project. This is simply my preferred approach.

I hope you found this guide helpful, let me know if you have any thoughts !

You can learn more about how I trained PurePastel on Gumroad where I provide detailed settings, thoughts, and comparisons. :)