Finetune workflows - Malware Dream B/C

First off I haven't released these on civit YET but am happy to share the links for you to test yourself - We're working on the NEXT Version with a larger dataset to sort of work off the third version.

The original inspirations that we've been trying to bake a better style for:

Version A - M4LW4R3 DR34M (MERGE)

https://civitai.com/models/62189/m4lw4r3-dr34m

(AKA NenekoSplat) HF Link: https://huggingface.co/Duskfallcrew/N3N3K0-Spl4T

Lora this is based from: https://civitai.com/models/61481/contrast

THE ITEMS THAT HAVE BEEN FINE TUNED:

Version B: (MALWARE DREAM VB)

https://huggingface.co/Duskfallcrew/malware-dream

This is fine tuned off one of the M4LW4R3 (Malware) DR34M lines

(aka Neneko Splat) - not the model that's on civit, but one of it's sub merges.

Version C: Malware Dream VC

https://huggingface.co/Duskfallcrew/malware-dream-v2

This is fine tuned off EPIC MIX V3's new 2.5D update which is here: https://civitai.com/models/20562/epic-mix-v3

Finding out that our madness in merging and lora binding CAN PRODUCE absolute MAJORLY amazing models that work well with loras AND fine tuning. Go us. We MAY make an update to v3 at some stage if it denotes needing it, we just said we weren't because V3 and V4 are roughly the same line but V3 stopped when we couldn't edit the model card.

WORKFLOW and Prompt testing for VERSION B

2D / CONTRAST STYLE

masterpiece, best quality,1girl, solo, long hair, looking at viewer, bangs, blue eyes, hair ornament, twintails,(zentangle, mandala, tangle, chaos, disorder, spider webs, entangle:0.6), jewelry, upper body, pink hair, heart, earrings, parted lips, choker, sailor collar, hair bun, parted bangs, makeup, lipstick, blue sailor collar, crescent, portrait, meme, circlet, sailor senshi uniform, heart choker, crescent earrings, sailor moon redraw challenge (meme), tsukino usagi, sailor moon

Negative prompt: (EasyNegative:1.0), (bad_prompt_version2:0.8), nsfw, open clothes, (low quality, worst quality:1.4), (bad anatomy), (inaccurate limb:1.2), bad composition, inaccurate eyes, extra digit, fewer digits, (extra arms:1.2) ng_deepnegative_v1_75t negprompt5 bad_prompt_version2

Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1900506326, Size: 512x768, Model hash: 69f83ad875, Model: Malware_Dream, Clip skip: 2, Version: v1.3.0, Hashes: {"vae": "df3c506e51"}

masterpiece, best quality, 1girl, solo, breasts, bangs, black hair, dress, (zentangle, mandala, tangle, chaos, disorder, spider webs, entangle:0.6), canimal ears, jewelry, upper body, wings, sky, choker, cat ears, necklace, hair bun, black dress, animal ear fluff, gun, double bun, makeup, black choker, cross, lipstick, angel wings, cross necklace, black lips

Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1515144859, Size: 512x768, Model hash: 69f83ad875, Model: Malware_Dream, Clip skip: 2, Version: v1.3.0, Hashes: {"vae": "df3c506e51"}

What Service Did you USE?

We used Last Ben's dreambooth via RUNPOD, you can access getting runpod set up here:

https://runpod.io/?ref=yx1lcptf

Rough Steps? UNET?

First version we just left it at the "DEFAULTS" for most of it, 2e6 I THINK ?

It doesn't mean we were CORRECT, in fact others have stated 5e6 works best for dreambooth finetuning. We'll be testing that for the third run today.

7500 steps for the main section, as we're not using the way LORA's do it (we haven't sorted out how that works entirely - so many ways of doing things so little time!).

THAT MEANS: UNET ENCODING STEPS IS 7500.

Usually text encoding has suggested values, I can't remember what ours was for Version B. We amped it up and then turned it down for Version C.

We had NO regularization images or CONCEPT images for Version B.

Version B has a CUTE style, but it largely retains the style of the original model.

And the original model was just a lora binding with some 2D style concepts, Contrast lora etc. I Think the recipie by now is in either the Neneko Splat or civit card.

What other styles for Version B?

masterpiece, best quality,(zentangle, mandala, tangle, entangle:0.6),1girl, solo, long hair, looking at viewer, skirt, gloves, holding, weapon,glasses, holding weapon, armor, blurry, blurry background, polearm, goggles, shoulder armor, gauntlets, eyewear on head, goggles on head, spear, breastplate, holding polearm,dynamic angle, side lighting, shiny skin, detailed eyes, detailed face,

Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 2016697571, Size: 512x768, Model hash: 69f83ad875, Model: Malware_Dream, Clip skip: 2, Version: v1.3.0, Hashes: {"vae": "df3c506e51"}

masterpiece, best quality,,(zentangle, mandala, tangle, entangle:0.6),solo, open mouth, simple background, 1boy, animal ears, jewelry, monochrome, male focus, earrings, cat ears, facial hair, facial mark, beard, purple theme, miqo'te

Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 3922721579, Size: 512x768, Model hash: 69f83ad875, Model: Malware_Dream, Clip skip: 2, Version: v1.3.0, Hashes: {"vae": "df3c506e51"}

What we learned a little bit about Version B

Being VERY careful with which models you fine tune from, it's not that Version B doesn't look cool - we're looking to make our own mark a little more, and Version B loses a little focus due to the Contrast LORA being in beta development stages. That lora had NO text encoding, and hilariously while it "WORKS" in theory - just mainly produces a 1-1 for Khit'li L'ocar - a character that we've made up for FFXIV.

More about the style though, we were STARTING TO TRY and come up with a silk screen almost Andy warhol inspired/Roy Lichtenstein style - Pop Art infused. We noticed, that you don't need to train a model to do this, you just need key prompts - or else sometimes it just gets a little confusing.

WORKFLOW and Prompt testing for VERSION C

2.5D + Contrast? - Different base model?

Prompts:

mksks style, Malware Dream,masterpiece, ((anime)), official art, best quality, (zentangle, mandala, tangle, entangle)animal ears, 1boy, solo, male focus, facial hair,red eyes , looking at viewer, beard, red hair, mustache, wings, lips,cat ears, denim jeans, leather jacket, ((detailed skin, skindentation, detailed eyes, detailed hair, detailed fabric, detailed background, detailed lighting, extremely detailed)), ((masterpiece, best quality, ultra-detailed, highres)), side lighting, rim lighting, lustrous skin,(bloom), (shine), lighting, ray tracing,((depth_of_field, rule of thirds, great composition,Dynamic angle)),(Highest picture quality), (Master's work), rich in detail, (highest quality), (best shadow),perfect lighting , perfect anatomy ,(masterpiece) fxaa, octane render,

Negative prompt: ((EasyNegative)), (bad_prompt_version2), ((nsfw, open clothes, nipple, naked, nude, cleavage)), ((animebadhands_v4)) negprompt5 bad-hands-3 bad-artist vile_prompt3, ((two heads, two people, extra people)), cropped, out of frame, text, watermark, logo , signature, artist name, twitter username, inaccurate eyes, extra digit, fewer digits, (extra arms:1.2) ng_deepnegative_v1_75t negprompt5 bad_prompt_version2

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 4243602853, Size: 512x768, Model hash: eb1cec1561, Model: Malware_Dream_v2, Denoising strength: 0.5, Clip skip: 2, Hires upscale: 2, Hires steps: 13, Hires upscaler: 8x_NMKD-Typescale_175k, Version: v1.3.0, Hashes: {"vae": "df3c506e51"}

mksks style, Malware Dream,masterpiece, ((anime)), official art, best quality, (zentangle, mandala, tangle, entangle), 1girl, solo focus,(( textured city skybreak background)), long hair, looking at viewer, bangs, blue eyes, shirt, long sleeves, blue hair, purple eyes, jacket, white shirt, upper body, food, hood, blunt bangs, sleeves past wrists, hoodie, mouth hold, hood down, blue jacket, blue nails, food in mouth, purple jacket, doughnut, ((fantasy, sci fi, ray tracing, octane render, fxaa, rtx)), ((masterpiece, best quality, ultra-detailed, highres)), side lighting, rim lighting, lustrous skin,(bloom), (shine), lighting, ray tracing,((depth_of_field, rule of thirds, great composition,Dynamic angle)),(Highest picture quality), (Master's work), rich in detail, (highest quality), (best shadow),perfect lighting , perfect anatomy ,(masterpiece) fxaa, octane render, BREAK

yonemiya mai style city background, extreme light and shadow, textured detail, realistic city background, best manga style, symmetry fantasy style art!! high detail hd 8k resolution trending pinterest dramatic light painted dark gloomy background contrast pastel colors no cropok night sky concept sketch 3-point lighting BREAK

(illustration:1.3),(extremely detailed:1.3),(highly detailed:1.2),(best quality:1.2),(masterpiece:1.2),(extremely detailed 8K wallpaper:1.2),(highres:1.3),(semi-realistic:1.1),(portraits:1.3),(anime:1.1),(cowboy shot:1.3), (solo:1.4),(perfect female body:1.3),(perfect anatomy:1.3),(wide hips:1.1),(thick thighs:1.1),(thigh gap:1.2),big butt,fat mons,(standing:1.3),(looking at viewer:1.4),(arms at sides:1.2),(corruption:1.4),

Negative prompt: ((EasyNegative)), (bad_prompt_version2), ((animebadhands_v4)) negprompt5 bad-hands-3 bad-artist vile_prompt3, ((two heads, two people, extra people)), cropped, out of frame, text, watermark, logo , signature, artist name, twitter username, inaccurate eyes, extra digit, fewer digits, (extra arms:1.2) ng_deepnegative_v1_75t negprompt5 bad_prompt_version2

Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 3239427637, Size: 512x768, Model hash: eb1cec1561, Model: Malware_Dream_v2, Denoising strength: 0.6, Clip skip: 2, Hires upscale: 2, Hires steps: 30, Hires upscaler: WaifuGAN_v3_30000, Version: v1.3.0, Hashes: {"vae": "df3c506e51"}

What's this MKSKS STYLE?

We're not SURE lol - we're not great at prompt engineering and sometimes get confused at what should and shouldn't be used. We heard this CAN be added in certain ways to "AMP UP" your prompt - but we're probably misreading that.

Your data doesn't have an instance tag now what?

We've HEARD from some places if you have a FILE name, it can go off that. Don't quote us on that one.

Contrast VS Style

You CAN train your model to do what we were HOPING to do, but it can get confusing for the end user. Some prompting works and negative prompting can - but the model did end up wilder depending on the prompt used.

The duotone/depth lighting DOES seem to look MUCH cooler on a 2.5D model than on a 2D model, and sort of makes a 2D model look even flatter in a much more lazy "bad art" way.

We've been trying to make it so you CAN still semi do a "MOE" look without it looking way too young. You can pull off tasteful kawaii without blurring the boundaries of what's legal and what's illegal in this case. Clearly this is prompting partially, but also trying to infer there's minimal data that can be used for ill gain in this instance.

Though, due to some of the data it CAN Prefer "SEXIER" more NSFW clothing. We're going to test this theory by re-training version C today and seeing what happens.

What SETTINGS FOR VERSION C?

Again similar to above, but this time we'll actually open up runpod and check as we think our tagging is done and we have to get the pictures uploaded! Also, don't mind if the settings are "NOT CORRECT" for fine tuning, we've just been playing around and seeing what we can do with other models. You ARE more than welcome with respect to let us know what we should be using in the comments. Do not take our settings with anything but a grain of salt, as we're not perfect at this side of the game.

First of all we're running it on a SECURE CLOUD instance, only using about 36 Cents an hour roughly. We have an automatic pod setup that lets us re-hash from the start.

Our set up:

1 x RTX A4500

12 vCPU 62 GB RAM

20 GB Disk

120 GB Network Volume

(USUALLY NVME because nerds)

The Settings:

UNet_Training_Steps= 7500

UNet_Learning_Rate= "2e-6"

Text_Encoder_Training_Steps= 450

Text_Encoder_Learning_Rate= "1e-6"

Text_Encoder_Concept_Training_Steps= 350

And then of course, offset noise is true, because even in an Anime model, who doesn't want offset noise?

All of this on THIS set up usually takes about 30 minutes to train. However long you spend TESTING IT IS ANOTHER PROBLEM!

So uh, i've done that now where do i put it?

Dreambooth Jupyter/Colab & Hugging Face Diffusers:

Ok so no matter WHAT set up you're using, last ben's notebook has an AUTOMATIC send to huggingface as it's own repository, sets it up to diffusers FOR YOU. The thing is, I don't know how to program so beyond that you're on your own!

When it's done uploading to your repository it will largely say something like this:

license: creativeml-openrail-m

tags:

- text-to-image

- stable-diffusion

---

### Malware_Dream_v2 Dreambooth model trained by Duskfallcrew with TheLastBen's fast-DreamBooth notebook

You'll need to edit it when you go in, and make sure your tags are set or else the API won't recognize your model is useable on the Hugging Face diffusers stuff.

I'm Curious! Can I beta test these when they're ready?

YES! We're looking to start trying to SLOW THE HELL DOWN and get more people to hang and do beta testing (and / or roast us for making hands that look like monsters.) - you don't really have to sign anything - we're not a company.

https://discord.gg/5t2kYxt7An

You WILL have to be ok with people using PLURAL KIT, as we're not about to change who we are, and while we understand that this is the internet and people don't listen - your choice is yours LOL. If you don't like people that are neurospicy, differently abled and weird? Don't join! simple as that!

We ONLY ask that you don't ON SELL anything or use for commercial use unless it's already released on civit.

We will be looking at adding people to a Huggingface TEAM so we can then upload testable CKPT/Safetensors without having to publicly open ourselves up to devastation and frustration.

Also: No doing illegal crap with anything we make sighs seriously.

We do recommend our new reddit where you can post content that you've made with our content or other forms of AI and other art:

https://www.reddit.com/r/earthndusk/

And that's a WRAP folks, we're off to go train VERSION D.... "DEEZ WHAT?" yea no, not deez nuts go away LOL.