Flux model training from just 1 image

They say that it's not the size of your dataset that matters. It's how you use it.

I have been doing some tests with single image (and few image) model trainings, and my conclusion is that this is a perfectly viable strategy depending on your needs.

A model trained on just one image may not be as strong as one trained on tens, hundreds or thousands, but perhaps it's all that you need.

What if you only have one good image of the model subject or style? This is another reason to train a model on just one image.

Single Image Datasets

The concept is simple. One image, one caption.

Since you only have one image, you may as well spend some time and effort to make the most out of what you have. So you should very carefully curate your caption.

What should this caption be? I still haven't cracked it, and I think Flux just gets whatever you throw at it. In the end I cannot tell you with absolute certainty what will work and what won't work.

Here are a few things you can consider when you are creating the caption:

Suggestions for a single image style dataset

1. Do you need a trigger word? For a style, you may want to do it just to have something to let the model recall the training. You may also want to avoid the trigger word and just trust the model to get it. For my style test, I did not use a trigger word.

2. Caption everything in the image.

3. Don't describe the style. At least, it's not necessary.

4. Consider using masked training (see Masked Training below).

Suggestions for a single image character dataset

1. Do you need a trigger word? For a character, I would always use a trigger word. This lets you control the character better if there are multiple characters.

For my character test, I did use a trigger word. I don't know how trainable different tokens are. I went with "GoWRAtreus" for my character test.

2. Caption everything in the image. I think Flux handles it perfectly as it is. You don't need to "trick" the model into learning what you want, like how we used to caption things for SD1.5 or SDXL (by captioning the things we wanted to be able to change after, and not mentioning what we wanted the model to memorize and never change, like if a character was always supposed to wear glasses, or always have the same hair color or style.

3. Consider using masked training (see Masked Training below).

Suggestions for a single image concept dataset

TBD. I'm not 100% sure that a concept would be easily taught in one image, that's something to test.

There's certainly more experimentation to do here. Different ranks, blocks, captioning methods.

If I were to guess, I think most combinations of things are going to produce good and viable results. Flux tends to just be okay with most things. It may be up to the complexity of what you need.

Masked training

This essentially means to train the image using either a transparent background, or a black/white image that acts as your mask. When using an image mask, the white parts will be trained on, and the black parts will not.

Note: I don't know how mask with grays, semi-transparent (gradients) works. If somebody knows, please add a comment below and I will update this.

Black & White masks or Transparent Background?

To be clear: When I trained my models for Flux, I simply erased the backgrounds of my images. I did not use a black/white mask. In the end it's the same thing, but it depends on the tool you are using. With Flux Comfy training, it wanted transparent PNGs so this is what I used.

What is it good for?

The benefits of training it this way is that we can focus on what we want to teach the model, and make it avoid learning things from the background, which we may not want.

If you instead were to cut out the subject of your training and put a white background behind it, the model will still learn from the white background, even if you caption it. And if you only have one image to train on, the model does so many repeats across this image that it will learn that a white background is really important. It's better that it never sees a white background in the first place

If you have a background behind your character, this means that your background should be trained on just as much as the character. It also means that you will see this background in all of your images. Even if you're training a style, this is not something you want. See images below.

Example without masking

I trained a model using only this image in my dataset.

The results can be found in this version of the model.

As we can see from these images, the model has learned the style and character design/style from our single image dataset amazingly! It can even do a nice bird in the style. Very impressive.

We can also unfortunately see that it's including that background, and a ton of small doll-like characters in the background. This wasn't desirable, but it was in the dataset. I don't blame the model for this.

Once again, with masking!

I did the same training again, but this time using a masked image:

It's the same image, but I removed the background in Photoshop. I did other minor touch-ups to remove some undesired noise from the image while I was in there.

The results can be found in this version of the model.

Now the model has learned the style equally well, but it never overtrained on the background, and it can therefore generalize better and create new backgrounds based on the art style of the character. Which is exactly what I wanted the model to learn.

The model shows signs of overfitting, but this is because I'm training for 2000 steps on a single image. That is bound to overfit.

How to create good masks

• You can use something like Inspyrnet-Rembg.

• You can also do it manually in Photoshop or Photopea. Just make sure to save it as a transparent PNG and use that.

• Inspyrnet-Rembg is also avaialble as a ComfyUI node.

Where can you do masked training?

I used ComfyUI to train my model. I think I used this workflow from CivitAI user Tenofas.

Note the "alpha_mask" setting on the TrainDatasetGeneralConfig.

Kohya_ss training tools also supports masked training.

There are also other trainers that utilizes masked training. I know OneTrainer supports it, but I don't know if their Flux training is functional yet or if it supports alpha masking.

I believe it is coming in kohya_ss as well.

If you know of other training scripts that support it, please write below and I can update this information.

It would be great if the option would be added to the CivitAI onsite trainer as well. With this and some simple "rembg" integration, we could make it easier to create single/few-image models right here on CivitAI.

Example Datasets & Models from single image training

Kawaii Style - failed first attempt without masks

Unfortunately I didn't save the captions I trained the model on. But it was automatically generated and it used a trigger word.

I trained this version of the model on the Shakker onsite trainer. They had horrible default model settings and if you changed them, the model still trained on the default settings so the model is huge (trained on rank 64).

As I mentioned earlier, the model learned the art style and character design reasonably well. It did however pick up the details from the background, which was highly undesirable. It was either that, or have a simple/no background. Which is not great for an art style model.

Kawaii Style - Masked training

An asian looking man with pointy ears and long gray hair standing. The man is holding his hands and palms together in front of him in a prayer like pose. The man has slightly wavy long gray hair, and a bun in the back. In his hair is a golden crown with two pieces sticking up above it. The man is wearing a large red ceremony robe with golden embroidery swirling patterns. Under the robe, the man is wearing a black undershirt with a white collar, and a black pleated skirt below. He has a brown belt. The man is wearing red sandals and white socks on his feet. The man is happy and has a smile on his face, and thin eyebrows.

Trained with 1x 1024x1024 image. Trained with 512x512 training settings.

The retraining with the masked setting worked really well. The model was trained for 2000 steps, and while there are certainly some overfitting happening, the results are pretty good throughout the epochs.

Please check out the models for additional images.

Overfitting and issues

This "successful" model does have overfitting issues. You can see details like the "horns/wings" at the top of the head of the dataset character appearing throughout images, even ones that don't have characters, like this one:

Funny if you know what they are looking for.

We can also see that even from early steps (250), body anatomy like fingers immediately break when the training starts.

I have no good solutions to this, and I don't know why it happens for this model, but not for the Atreus one below.

Maybe it breaks if the dataset is too cartoony, until you have trained it for enough steps to fix it again?

If anyone has any anecdotes about fixing broken flux training anatomy, please suggest solutions in the comments.

Character - God of War Ragnarok: Atreus - Single image, rank16, 2000 steps

A youthful warrior, GoWRAtreus is approximately 14 years old, stands with a focused expression. His eyes are bright blue, and his face is youthful but hardened by experience. His hair is shaved on the sides with a short reddish-brown mohawk. He wears a yellow tunic with intricate red markings and stitching, particularly around the chest and shoulders. His right arm is sleeveless, exposing his forearm, which is adorned with Norse-style tattoos. His left arm is covered in a leather arm guard, adding a layer of protection. Over his chest, crossed leather straps hold various pieces of equipment, including the fur mantle that drapes over his left shoulder. In the center of his chest, a green pendant or accessory hangs, adding a touch of color and significance. Around his waist, a yellow belt with intricate patterns is clearly visible, securing his outfit. Below the waist, his tunic layers into a blue skirt-like garment that extends down his thighs, over which tattered red fabric drapes unevenly. His legs are wrapped in leather strips, leading to worn boots, and a dagger is sheathed on his left side, ready for use.

Trained with 1x 512x1024 image. Trained with 512x512 training settings.

After the success of the single image Kawaii style, I knew I wanted to try this single image method with a character.

I trained the model for 2000 steps, but I found that the model was grossly overfit (more on that below). I tested earlier epochs and found that the earlier epochs, at 250 and 500 steps, were actually the best. They had learned enough of the character for me, but did not overfit on the single front-facing pose.

This model was trained at Network Dimension and Alpha (Network rank) 16.

Above: The model severely overfit at 2000 steps.

Above: The model producing decent results at 250 steps.

An additional note worth mentioning is that the 2000 step version was actually almost usable at 0.5 weight. So even though the model is overfit, there may still be something to salvage inside.

Character - God of War Ragnarok: Atreus - 4 images, rank16, 2000 steps

I also trained a version using 4 images from different angles (same pose).

Trained with 4 images. The images were 1024x1024, 512x1024, 384x1024, 768x1024. Trained with 512x512 training settings.

This version was a bit more poseable at higher steps. It was a lot easier to get side or back views of the character without going into really high weights.

The model had about the same overfitting problems when I used the 2000 step version, and I found the best performance at step ~250-500.

This model was trained at Network Dimension and Alpha (Network rank) 16.

Character - God of War Ragnarok: Atreus - Single image, rank16, 400 steps, rank4

I decided to re-train the single image version at a lower Network Dimension and Network Alpha rank. I went with rank 4 instead. And this worked just as well as the first model. I trained it on max steps 400, and below I have some random images from each epoch.

Link to full size image

Trained with 1x 512x1024 image. Trained with 512x512 training settings.

It does not seem to overfit at 400, so I personally think this is the strongest version. It's possible that I could have trained it on more steps without overfitting at this network rank.

Signs of overfitting

I'm not 100% sure about this, but I think that Flux looks like this when it's overfit.

Fabric / Paper Texture

We can see some kind of texture that reminds me of rough fabric. I think this is just noise that is not getting denoised properly during the diffusion process.

Fuzzy Edges

We can also observe fuzzy edges on the subjects in the image. I think this is related to the texture issue as well, but just in small form.

Ghosting

We can also see additional edge artifacts in the form of ghosting. It can cause additional fingers to appear, dual hairlines, and general artifacts behind objects.

All of the above are likely caused by the same thing. These are the larger visual artifacts to keep an eye out for. If you see them, it's likely the model has a problem.

For smaller signs of overfitting, lets continue below.

Finding the right epoch

If you keep on training, the model will inevitebly overfit.

One of the key things to watch out for when training with few images, is to figure out where the model is at its peak performance.

• When does it give you flexibility while still looking good enough?

The key to this is obviously to focus more on epochs, and less on repeats. And making sure that you save the epochs so you can test them.

You then want to do run X/Y grids to find the sweet spot.

I suggest going for a few different tests:

1. Try with the originally trained caption

Use the exact same caption, and see if it can re-create the image or get a similar image. You may also want to try and do some small tweaks here, like changing the colors of something.

If you used a very long and complex caption, like in my examples above, you should be able to get an almost replicated image. This is usually called memorization or overfitting and is considered a bad thing. But I'm not so sure it's a bad thing with Flux. It's only a bad thing if you can ONLY get that image, and nothing else.

If you used a simple short caption, you should be getting more varied results.

2. Test the model extremes

If it was of a character from the front, can you get the back side to look fine or will it refuse to do the back side? Test it on things it hasn't seen but you expect to be in there.

3. Test the model's flexibility

If it was a character, can you change the appearance? Hair color? Clothes? Expression? If it was a style, can it get the style but render it in watercolor?

4. Test the model's prompt strategies

Try to understand if the model can get good results from short and simple prompts (just a handful of words), to medium length prompts, to very long and complex prompts.

Note: These are not Flux exclusive strategies. These methods are useful for most kinds of model training. Both images and also when training other models.

Drawbacks & Weaknesses

I need to explore this further, but it seems like these 400 steps weren't enough for the model to learn the face. Not at this size of the image used at least. Perhaps one face image and one body image could be enough? That would be interesting to try for a future experiment.

More data is needed.

Key Learning: Iterative Models (Synthetic data)

One thing you can do is to use a single image trained model to create a larger dataset for a stronger model.

It doesn't have to be a single image model of course, this also works if you have a bad initial dataset and your first model came out weak or unreliable.

It is possible that with some luck, you're able to get a few good images to to come out from your model, and you can then use these images as a new dataset to train a stronger model.

This is how these series of Creature models were made:

https://civitai.com/models/378882/arachnid-creature-concept-sd15

https://civitai.com/models/378886/arachnid-creature-concept-pony

https://civitai.com/models/378883/arachnid-creature-concept-sdxl

https://civitai.com/models/710874/arachnid-creature-concept-flux

The first version was trained on a handful of low quality images, and the resulting model got one good image output in 50. Rinse and repeat the training using these improved results and you eventually have a model doing what you want.

I have an upcoming article on this topic as well. If it interests you, maybe give a follow and you should get a notification when there's a new article.

Flux Lite Training on CivitAI

If you think it would be good to have the option of training a smaller, faster, cheaper LoRA here at CivitAI, please check out this "petition/poll/article" about it and give it a thumbs up to gauge interest in something like this.

Cool Beans! I learned today!

That's great! I'm happy that you learned something new!

If you appreciated the article, feel free to drop some ⚡⚡⚡ on the article, or perhaps download one of the models to get something from it. Since 1000 buzz is $1 and I gave you my two cents on the subject, it's only fair that you put in 20 buzz in return? :D

Alternatively, engage in the comments below or write your own article about your experiments and findings, and link them below!

Thanks for reading! You are a star!