Verified: 10 months ago
SafeTensor
The FLUX.1 [dev] Model is licensed by Black Forest Labs. Inc. under the FLUX.1 [dev] Non-Commercial License. Copyright Black Forest Labs. Inc.
IN NO EVENT SHALL BLACK FOREST LABS, INC. BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH USE OF THIS MODEL.
Trigger word: juno
Optional Triggers: space helmet, overboots, blaster
The optional trigger will only show up when prompted for.
Does not currently work with the onsite generator
(civitai does not yet support LoKr)
----------
HOW WAS THIS MADE?
----------
This is a LoKr trained on 115 images of juno. (about 40% ingame screenshots, 40% renders I made myself from a t-posing juno, and 20% official concept art)
This was trained on SimpleTuner for 5400 steps, at batch 6, on an A6000
Training strategy:
So the point here was to train a genuinely flexible lora (LoKr) using nothing but repetitive "meh" quality data.
I limited myself to an ingame screenshots, self-made blender renders & official concept art for my dataset.
Dataset examples:
In addition to that, I've noticed that while multi-resolution training is important and needed, it isn't needed throughout the entire training run
So my theory was that I can just purposely train 50% of my total steps at a resolution of 512px. then, once it's adapted well enough, stop training, add in the 768px and 1024px datasets, and resume training for the final 3k steps. (albeit much slower, due to the increased size of the images)
The result?
a huge success!
There is no decrease in quality nor flexibility from my other LoKr tests, and more than that, it trained significantly faster!
Captioning Strategy:
concept art, juno3D rendering, space helmet, juno, t-pose, overboots3D rendering, juno, overbootsingame screenshot, blasteringame screenshot of juno, blaster, space helmet, overboots
Basically, mention the style and what is visible. But dont go into too much detail. Flux can extrapolate A LOT of info. So only mention things that will be misunderstood, or things that you are specifically training and want to be able to prompt for explicitly.
lycoris settings:
{
"algo": "lokr",
"multiplier": 1.0,
"linear_dim": 10000,
"linear_alpha": 1,
"factor": 12,
"apply_preset": {
"target_module": [
"Attention",
"FeedForward"
],
"module_algo_map": {
"Attention": {
"factor": 12
},
"FeedForward": {
"factor": 6
}
}
}
}config.env:
RESUME_CHECKPOINT='latest'
DATALOADER_CONFIG='config/juno.json'
ASPECT_BUCKET_ROUNDING='2'
TRAINING_SEED='42'
USE_EMA='false'
USE_XFORMERS='false'
MINIMUM_RESOLUTION='0'
OUTPUT_DIR='output/juno'
USE_DORA='false'
USE_BITFIT='false'
LORA_TYPE='lycoris'
LYCORIS_CONFIG='config/lycoris_config.json'
PUSH_TO_HUB='false'
PUSH_CHECKPOINTS='false'
MAX_NUM_STEPS='10000'
NUM_EPOCHS='0'
CHECKPOINTING_STEPS='100'
CHECKPOINTING_LIMIT='40'
HUB_MODEL_NAME='simpletuner-lora'
DEBUG_EXTRA_ARGS=''
MODEL_TYPE='lora'
MODEL_NAME='black-forest-labs/FLUX.1-dev'
FLUX='true'
KOLORS='false'
STABLE_DIFFUSION_3='false'
STABLE_DIFFUSION_LEGACY='false'
TRAIN_BATCH_SIZE='4'
USE_GRADIENT_CHECKPOINTING='true'
GRADIENT_ACCUMULATION_STEPS='1'
CAPTION_DROPOUT_PROBABILITY='0'
RESOLUTION_TYPE='pixel_area'
RESOLUTION='1024'
VALIDATION_SEED='42'
VALIDATION_STEPS='100'
VALIDATION_RESOLUTION='1024x1024'
VALIDATION_GUIDANCE='3.5'
VALIDATION_GUIDANCE_RESCALE='0.0'
VALIDATION_NUM_INFERENCE_STEPS='20'
VALIDATION_PROMPT='concept art of juno, space helmet, white background'
ALLOW_TF32='true'
MIXED_PRECISION='bf16'
OPTIMIZER=optimi-lion
LEARNING_RATE='5e-5'
LR_SCHEDULE='polynomial'
LR_WARMUP_STEPS='100'
BASE_MODEL_PRECISION='int8-quanto'
ACCELERATE_EXTRA_ARGS=''
TRAINING_NUM_PROCESSES='1'
TRAINING_NUM_MACHINES='1'
VALIDATION_TORCH_COMPILE='false'
TRAINER_DYNAMO_BACKEND='no'
TRAINER_EXTRA_ARGS='--lr_end=1e-8 --compress_disk_cache --optimizer_config=weight_decay=1e-3'Dataset config for first 3k steps:
[
{
"id": "juno-512",
"type": "local",
"instance_data_dir": "/mnt/md/0/DATASET/juno",
"crop": false,
"resolution": 512,
"resolution_type": "pixel_area",
"minimum_image_size": 512,
"prepend_instance_prompt": false,
"instance_prompt": "",
"only_instance_prompt": false,
"caption_strategy": "textfile",
"cache_dir_vae": "/mnt/md/0/AI/SimpleTuner/SimpleTuner-Cache/vae-cache/juno-512",
"probability": 1.0,
"repeats": 2,
"preserve_data_backend_cache": true,
"cache_file_suffix": "-juno-512"
},
{
"id": "juno-text-embeds",
"type": "local",
"dataset_type": "text_embeds",
"default": true,
"cache_dir": "/mnt/md/0/AI/SimpleTuner/SimpleTuner-Cache/embed-cache/juno",
"disabled": false,
"write_batch_size": 128
}
]Dataset config for steps 3000~5400:
[
{
"id": "juno-1024",
"type": "local",
"instance_data_dir": "/mnt/md/0/DATASET/juno",
"crop": false,
"resolution": 1024,
"resolution_type": "pixel_area",
"minimum_image_size": 1024,
"prepend_instance_prompt": false,
"instance_prompt": "",
"only_instance_prompt": false,
"caption_strategy": "textfile",
"cache_dir_vae": "/mnt/md/0/AI/vae-cache/juno-1024",
"probability": 1.0,
"repeats": 1,
"preserve_data_backend_cache": true,
"cache_file_suffix": "-juno-1024"
},{
"id": "juno-768",
"type": "local",
"instance_data_dir": "/mnt/md/0/DATASET/juno",
"crop": false,
"resolution": 768,
"resolution_type": "pixel_area",
"minimum_image_size": 768,
"prepend_instance_prompt": false,
"instance_prompt": "",
"only_instance_prompt": false,
"caption_strategy": "textfile",
"cache_dir_vae": "/mnt/md/0/AI/vae-cache/juno-768",
"probability": 1.0,
"repeats": 2,
"preserve_data_backend_cache": true,
"cache_file_suffix": "-juno-768"
},{
"id": "juno-512",
"type": "local",
"instance_data_dir": "/mnt/md/0/DATASET/juno",
"crop": false,
"resolution": 512,
"resolution_type": "pixel_area",
"minimum_image_size": 512,
"prepend_instance_prompt": false,
"instance_prompt": "",
"only_instance_prompt": false,
"caption_strategy": "textfile",
"cache_dir_vae": "/mnt/md/0/AI/vae-cache/juno-512",
"probability": 1.0,
"repeats": 2,
"preserve_data_backend_cache": true,
"cache_file_suffix": "-juno-512"
},
{
"id": "juno-text-embeds",
"type": "local",
"dataset_type": "text_embeds",
"default": true,
"cache_dir": "/mnt/md/0/AI/embed-cache/juno",
"disabled": false,
"write_batch_size": 128
}
]