Juno for flux (by caith) - Overwatch 2

Trigger word: juno

Optional Triggers: space helmet, overboots, blaster

The optional trigger will only show up when prompted for.

Does not currently work with the onsite generator

(civitai does not yet support LoKr)
----------

HOW WAS THIS MADE?
----------

This is a LoKr trained on 115 images of juno. (about 40% ingame screenshots, 40% renders I made myself from a t-posing juno, and 20% official concept art)

This was trained on SimpleTuner for 5400 steps, at batch 6, on an A6000

Training strategy:

So the point here was to train a genuinely flexible lora (LoKr) using nothing but repetitive "meh" quality data.

I limited myself to an ingame screenshots, self-made blender renders & official concept art for my dataset.

Dataset examples:

In addition to that, I've noticed that while multi-resolution training is important and needed, it isn't needed throughout the entire training run

So my theory was that I can just purposely train 50% of my total steps at a resolution of 512px. then, once it's adapted well enough, stop training, add in the 768px and 1024px datasets, and resume training for the final 3k steps. (albeit much slower, due to the increased size of the images)

The result?
a huge success!

There is no decrease in quality nor flexibility from my other LoKr tests, and more than that, it trained significantly faster!

Captioning Strategy:

1. concept art, juno

2. 3D rendering, space helmet, juno, t-pose, overboots

3. 3D rendering, juno, overboots

4. ingame screenshot, blaster

5. ingame screenshot of juno, blaster, space helmet, overboots

Basically, mention the style and what is visible. But dont go into too much detail. Flux can extrapolate A LOT of info. So only mention things that will be misunderstood, or things that you are specifically training and want to be able to prompt for explicitly.

lycoris settings:

{
    "algo": "lokr",
    "multiplier": 1.0,
    "linear_dim": 10000,
    "linear_alpha": 1,
    "factor": 12,
    "apply_preset": {
        "target_module": [
            "Attention",
            "FeedForward"
        ],
        "module_algo_map": {
            "Attention": {
                "factor": 12
            },
            "FeedForward": {
                "factor": 6
            }
        }
    }
}

config.env:

RESUME_CHECKPOINT='latest'
DATALOADER_CONFIG='config/juno.json'
ASPECT_BUCKET_ROUNDING='2'
TRAINING_SEED='42'
USE_EMA='false'
USE_XFORMERS='false'
MINIMUM_RESOLUTION='0'
OUTPUT_DIR='output/juno'
USE_DORA='false'
USE_BITFIT='false'
LORA_TYPE='lycoris'
LYCORIS_CONFIG='config/lycoris_config.json'
PUSH_TO_HUB='false'
PUSH_CHECKPOINTS='false'
MAX_NUM_STEPS='10000'
NUM_EPOCHS='0'
CHECKPOINTING_STEPS='100'
CHECKPOINTING_LIMIT='40'
HUB_MODEL_NAME='simpletuner-lora'
DEBUG_EXTRA_ARGS=''
MODEL_TYPE='lora'
MODEL_NAME='black-forest-labs/FLUX.1-dev'
FLUX='true'
KOLORS='false'
STABLE_DIFFUSION_3='false'
STABLE_DIFFUSION_LEGACY='false'
TRAIN_BATCH_SIZE='4'
USE_GRADIENT_CHECKPOINTING='true'
GRADIENT_ACCUMULATION_STEPS='1'
CAPTION_DROPOUT_PROBABILITY='0'
RESOLUTION_TYPE='pixel_area'
RESOLUTION='1024'
VALIDATION_SEED='42'
VALIDATION_STEPS='100'
VALIDATION_RESOLUTION='1024x1024'
VALIDATION_GUIDANCE='3.5'
VALIDATION_GUIDANCE_RESCALE='0.0'
VALIDATION_NUM_INFERENCE_STEPS='20'
VALIDATION_PROMPT='concept art of juno, space helmet, white background'
ALLOW_TF32='true'
MIXED_PRECISION='bf16'
OPTIMIZER=optimi-lion
LEARNING_RATE='5e-5'
LR_SCHEDULE='polynomial'
LR_WARMUP_STEPS='100'
BASE_MODEL_PRECISION='int8-quanto'
ACCELERATE_EXTRA_ARGS=''
TRAINING_NUM_PROCESSES='1'
TRAINING_NUM_MACHINES='1'
VALIDATION_TORCH_COMPILE='false'
TRAINER_DYNAMO_BACKEND='no'
TRAINER_EXTRA_ARGS='--lr_end=1e-8 --compress_disk_cache --optimizer_config=weight_decay=1e-3'

Dataset config for first 3k steps:

[
    {
        "id": "juno-512",
        "type": "local",
        "instance_data_dir": "/mnt/md/0/DATASET/juno",
        "crop": false,
        "resolution": 512,
        "resolution_type": "pixel_area",
        "minimum_image_size": 512,
        "prepend_instance_prompt": false,
        "instance_prompt": "",
        "only_instance_prompt": false,
        "caption_strategy": "textfile",
        "cache_dir_vae": "/mnt/md/0/AI/SimpleTuner/SimpleTuner-Cache/vae-cache/juno-512",
        "probability": 1.0,
        "repeats": 2,
        "preserve_data_backend_cache": true,
        "cache_file_suffix": "-juno-512"
    },
    {
        "id": "juno-text-embeds",
        "type": "local",
        "dataset_type": "text_embeds",
        "default": true,
        "cache_dir": "/mnt/md/0/AI/SimpleTuner/SimpleTuner-Cache/embed-cache/juno",
        "disabled": false,
        "write_batch_size": 128
  }
]

Dataset config for steps 3000~5400:

[
        {
        "id": "juno-1024",
        "type": "local",
        "instance_data_dir": "/mnt/md/0/DATASET/juno",
        "crop": false,
        "resolution": 1024,
        "resolution_type": "pixel_area",
        "minimum_image_size": 1024,
        "prepend_instance_prompt": false,
        "instance_prompt": "",
        "only_instance_prompt": false,
        "caption_strategy": "textfile",
        "cache_dir_vae": "/mnt/md/0/AI/vae-cache/juno-1024",
        "probability": 1.0,
        "repeats": 1,
        "preserve_data_backend_cache": true,
        "cache_file_suffix": "-juno-1024"
    },{
        "id": "juno-768",
        "type": "local",
        "instance_data_dir": "/mnt/md/0/DATASET/juno",
        "crop": false,
        "resolution": 768,
        "resolution_type": "pixel_area",
        "minimum_image_size": 768,
        "prepend_instance_prompt": false,
        "instance_prompt": "",
        "only_instance_prompt": false,
        "caption_strategy": "textfile",
        "cache_dir_vae": "/mnt/md/0/AI/vae-cache/juno-768",
        "probability": 1.0,
        "repeats": 2,
        "preserve_data_backend_cache": true,
        "cache_file_suffix": "-juno-768"
    },{
        "id": "juno-512",
        "type": "local",
        "instance_data_dir": "/mnt/md/0/DATASET/juno",
        "crop": false,
        "resolution": 512,
        "resolution_type": "pixel_area",
        "minimum_image_size": 512,
        "prepend_instance_prompt": false,
        "instance_prompt": "",
        "only_instance_prompt": false,
        "caption_strategy": "textfile",
        "cache_dir_vae": "/mnt/md/0/AI/vae-cache/juno-512",
        "probability": 1.0,
        "repeats": 2,
        "preserve_data_backend_cache": true,
        "cache_file_suffix": "-juno-512"
    },
    {
        "id": "juno-text-embeds",
        "type": "local",
        "dataset_type": "text_embeds",
        "default": true,
        "cache_dir": "/mnt/md/0/AI/embed-cache/juno",
        "disabled": false,
        "write_batch_size": 128
  }
]