SDXL with FLUX

This is a simple idea for people who can run FLUX locally or otherwise have full access to img2img, and especially to those who have less than 24gb vram and like to experiment. As far as I know, it doesn't work on Civitai because you can't switch from an SDXL model to FLUX using image-to-image variations. At least, I couldn't find a way when I tried - please let me know if I'm wrong about this.

I've been making the initial image with txt2img in SDXL and producing the final image with FLUX through img2img. This can also apply to SD1 and Pony in place of SDXL. I thought it might be interesting to show a demonstration for those who haven't tried it.

For reference, I'm using a computer with 16gb vram (nvidia 4060), 32 gb ram, and running FLUX in WebforgeUI.

Advantages of including SDXL:

• It's significantly faster to generate multiple images to have a selection to pick the best style and composition.

• Generating through img2img with FLUX can be faster than txt2img at lower denoise values with the same number of steps.

• It's possible to use SDXL's vast library of style LoRAs in txt2img, then FLUX keeps a similar style in img2img without the LoRA.

Advantages of including FLUX:

• It can add or fix text.

• It can fix anatomy and structure problems.

• It adds extra fidelity, details, and smooths out the linework.

• It can be satisfying seeing FLUX's transformations on what SDXL was struggling with.

Examples

Here are some before and after images I made using this method, along with the prompts and settings I used in img2img.

Cafe Chroma

Here I made the top text say "Cafe Chroma" and the small sign on the wall say "WELCOME". Flux also smoothed out and added small details in areas like the potted plants.

illustration of the exterior of a cafe, text "Cafe Chroma". Table and chairs, bicycle, and potted plants on the sidewalk. Small sign with text "WELCOME".
Steps: 30, Sampler: Euler, Schedule type: Simple, CFG scale: 1, Distilled CFG Scale: 3.5, Seed: 782544532, Size: 896x1152, Model hash: bea01d51bd, Model: flux1-dev-bnb-nf4-v2, Denoising strength: 0.4

link to full sizes

The Lute Player

Here the main objective was to make the woman's hands look better.

photograph of an elegant woman with delicate features playing a lute
Steps: 30, Sampler: Euler, Schedule type: Simple, CFG scale: 1, Distilled CFG Scale: 3.5, Seed: 353199320, Size: 896x1152, Model hash: bea01d51bd, Model: flux1-dev-bnb-nf4-v2, Denoising strength: 0.4

link to full sizes

Magical Library

Here I used FLUX to smooth out the details for a cleaner look and make the designs look more symmetrical.

victorian style magical library made of gold, lamp light, symmetrical golden decorations and embellishments, books, style of art nouveau, art deco
Steps: 30, Sampler: Euler, Schedule type: Simple, CFG scale: 1, Distilled CFG Scale: 3.5, Seed: 3996458741, Size: 896x1152, Model hash: bea01d51bd, Model: flux1-dev-bnb-nf4-v2, Denoising strength: 0.5

link to full sizes

Living Room Illustration

Here FLUX tidied up the room and made it easier to see what the small objects are supposed to be.

illustration of a living room
Steps: 30, Sampler: Euler, Schedule type: Simple, CFG scale: 1, Distilled CFG Scale: 3.5, Seed: 1069368321, Size: 896x1152, Model hash: bea01d51bd, Model: flux1-dev-bnb-nf4-v2, Denoising strength: 0.4

link to full sizes

Steps

1. Generate an image with an SDXL checkpoint plus any LoRAs.

2. Switch to a FLUX model

3. Put the image in img2img and set the denoise value to around 3 to 5 (adjust up or down as needed). I typically use between 20 and 30 steps.

4. You may need to write a new prompt explaining exactly what you want to keep or change in the image.

5. Generate and tweak until you get an image you're happy with.