I hope that this article piques your interest and that you'll read the whole thing, even if you feel like you already were aware that changing one setting can affect the whole pipelline.
I have been prompting for about 1-2 years now and I still find ways to surprise myself.
I knew that it was important to keep your settings the same when testing, but I wasn't this aware about it.
You may have heard that you shouldn't just copy someones prompt and use it. That is mostly true. If you can match their settings exactly or close enough, you can often get pretty close with a word for word prompt, same seed, same cfg etc.
But have you ever tried taking someone's prompt and then used your own favored sampler/scheduler/model then tried to upscale a lores prompt?
Have you noticed that sometimes a low-res prompt just isn't your result when you upscale?
This article explores the importance of all of that, and the prompting strategy used here is to wait for the final image to finish upscaling to your chosen resolution. And then pick 1 word at a time that I feel doesn't belong in the image.
I started from a 0-0 positive/negative prompt then added one negative word at a time to transition from start to end... to see my whole process please see this companion article to show how to create a "base prompt" which you can use when using the same scheduler/samper/model/resolution/seed ETC.
Once you're satisfied with the quality images from a single word: in this case - turtle --- before you start adding more negative prompts you'll want to take note of your base prompt, as well as all the settings you used: sampler, scheduler, model, #steps + #steps hires, CFG scale, denoising rate (I used 0.6 for this test), etc.
At least the standard A1111 prompt editor only saves your positive/negative prompt settings, but going forward this should revolutionize the way you think of prompting and should shape your future prompt strategies.
How Aware are You about Prompting?
Did you know that a different scheduler can throw off your results?
Did you know if you mixed and matched settings/prompts and models/schedulers between prompting that it affects your final image and the portability of your final prompt?
Do you usually start from a prompt base, then use that between models? Between schedulers? Between samplers?
One thing that I feel is a key take away from this is simple:
one word can change it all
When creating a prompt that can truly go from one model to the next without totally destroying the prompt (or the resulting image)... it's important to go one word a time and monitor the results closely.
Keep in mind that if you're trying to create a prompt that's tuned for hires upscales, to craft your prompt after hires upscaling each image for each added prompt.
I've also discovered that when crafting a tuned prompt for a scheduler/model that when you switch schedulers/models/samplers that it won't have a desirable result. The below images are the best images after starting from one model, and then going into the next one, then back and forth until we get a similar image created in both.
I did this through the use of 1 positive prompt and the rest negative prompts that were tuned to the model that I was working in.
Config Settings: (see attachments)
This config is for my current work in progress scheduler which has not been updated to Github yet.
Keep in mind that for this test we are using a hybrid blend scheduler, simple_kes , and using a custom configuration (see attached for detailed config).
In our scheduler models we are using two newly designed schedulers I've called "euler" and "euler_advanced" which uses advanced formulas in the scheduling logic which I felt would be used if euler was a scheduler and not just a sampler. The formulas are designed with euler in mind, though for this test, we use the sampler DPM++2M.
Step 0:
Using only negative prompts to guide the space, I created up to here with negative prompts:
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 86758142da, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "86758142da"}
Let's swap schedulers for our base prompt!
And the same exact prompt and settings with using a different scheduler:
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 86758142da, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "86758142da"}
This shows that although nothing changed except the scheduler, that...
tuning prompts require same settings - including the scheduler
...a prompt tuned for one will change when swapping schedulers.
I'll pause periodically in our test to show how swapping to a different scheduler while tuning your prompt will result in negative desired images.
All other image settings remained the same throughout the test.
Notes: This test ran with these base settings to start.
schedule designed with: Karras Exponential
model designed with: FusionX-Realistic_v3_float16
base prompt:
ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone,
Step 1:
Add a positive prompt then one at a time, add negative prompts after it resolves to a hires image.
turtle,
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone, extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 86758142da, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "86758142da"}
A Planned Pause
Pausing here, let's try the same prompt crafted for this scheduler, and this model, and let's try using a different scheduler, just to see the results:
turtle,
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone, extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 86758142da, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "86758142da"}
In this we show that a prompt is scheduler specific.
When you use a different scheduler which transposes noise differently, you're bound to get a different result. So even though nothing changed except swapping a scheduler, it makes ALL the difference.
Continuing with our test...
Starting with the FusionX-Realistic_v3_float16 , we added the word "turtle" to our positive prompt, and to our negative prompt: "extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper".
The base prompt was designed with this model, and this schedule type.
Step 2:
Switch to the other model and add more prompts to get to a good switch image
turtle,
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone, extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper, painting, magazine, giant_turtle, snout, armadillo, ball,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 5dc2d95ed5, Model: ACRBasev3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "5dc2d95ed5"}
Swapping to ACRBasev3_float16, we added painting, magazine, giant_turtle, snout, armadillo, ball
Another planned pause
Let's take another break, and see how this might change with another scheduler.
turtle,
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone, extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper, painting, magazine, giant_turtle, snout, armadillo, ball,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 5dc2d95ed5, Model: ACRBasev3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "5dc2d95ed5"}
As shown in the previous pause, when using a different scheduler you will get different results. Even when just changing the scheduler.
Continuing back to our test...
Step 3:
Switch back, keeping the same image settings used with the other model, unless a rollback of prompts is needed.
turtle,
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone, extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper, painting, magazine, giant_turtle, snout, armadillo, ball, frog, snake,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 86758142da, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "86758142da"}
Swapping to FusionX-Realistic_v3_float16 we added frog, snake,
Step 4:
Switch back, keeping the same image settings used with the other model, unless a rollback of prompts is needed.
turtle,
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone, extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper, painting, magazine, giant_turtle, snout, armadillo, ball, helmet, pixel art,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 5dc2d95ed5, Model: ACRBasev3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "5dc2d95ed5"}
Swapping back to: ACRBasev3_float16, we added "helmet, pixel art"
Let me just pause in this image run to say that we had to roll back some changes and try something else.
frog, snake --> helmet, pixel art,
Step 5:
Switch back, keeping the same image settings used with the other model, unless a rollback of prompts is needed.
a large turtle, grassy background,
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone, extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper, painting, magazine, giant_turtle, snout, armadillo, ball, helmet, pixel art, slime,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 86758142da, Model: FusionX-Realistic_v3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "86758142da"}
Swapping back to FusionX-Realistic_v3_float16, we appended "a large" to turtle, and added grassy background to positive prompts, and added "slime"
Step 6:
Switch back, keeping the same image settings used with the other model, unless a rollback of prompts is needed.
turtle, grassy background,
Negative prompt: ghost, pose:::covered_body!!, hand_details:::polished_nails, painted_nails!!, blurry, iphone, extra_legs, underwater, extra_turtle, object_ontop_shell, extra_tail, flipper, painting, magazine, giant_turtle, snout, armadillo, ball, helmet, pixel art, slime, tongue with closed_mouth,
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras Exponential, CFG scale: 8, Seed: 88120186151, Size: 640x720, Model hash: 5dc2d95ed5, Model: ACRBasev3_float16, Denoising strength: 0.6, Clip skip: 2, Hypertile U-Net: True, Hires upscale: 2, Hires steps: 35, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.10.1, Hashes: {"model": "5dc2d95ed5"}
This last picture, we rolled back "large" and added "tongue with closed_mouth"
Summary
Once you get to a spot where you think it's the best you can do in that model, swap over to the other model and test it out. Then refine some more. Then swap back to the other model. rinse and repeat until its done. I did 6 swaps back and forth until the final two images with additional positive prompts.
As a case-in-point, I did test the prompt with a different scheduler a few times. And this is what I discovered:
Prompts tuned for a model, and a schedule matters!
In addition to all the rest keeping them the same settings for cfg, sampler, steps, hires steps, ETC: matters!