What to expect from standard Stable Diffusion tunables

I tried to learn SD the same way I learn about any new unexplained phenomenon: I poke at it experimentally and see what happens.

Here are my Lab Notes (Queue HS Science montage…)

• NB: Some items are in more than one category. That is usually an indicator of “small value change, small effect. Large value change, large effect”

• pos:"woman, sun dress,smiling, garden", neg:"text, watermark"

BROAD CONTENT CHANGE

(eg: change of art style, AND/OR change of subject completely)

• Changing the CFG value
  
  This is randomly more or less random, in some ways. For example, if you do a sliding check from small values to large, it can do things like add in people half way… but then further on, remove them. This shows CFG=8 vs CFG=7
  
  

• Changing base model
  
  Some base models act like a LORA, and dramatically change even the style of the art. This can often happen even if you explicitly specify the style you actually want in the prompt. The biggest example of this is moving a prompt from a SD1.5 model to a SDXL model. It does really poorly at anime style unless you use particular phrasing. At the same time, some models share a lot in common with other ones, so certain ones may not be so different.
  
  

• Changing ANYTHING in the prompt. Positive, or negative
  

  • Changing literally anything in the prompt, even if at a language level there is no difference, can and usually does radically alter the result.

  • Even if you add a negative that is not present to begin with.

  • Even if you add or remove a comma at the end with nothing after it.
    For this sample, I changed the negative from “text, watermark” to just “watermark”
    
    

• Changing Sampler Type ::  SDXL SPECIAL CASE
  For some reason, if you want particular art styles from the SDXL base.. you may have to jump around Sampler Types.
  As a specific example: if you set “Anime” as your first prompt item for SDXL 1.0 Base, the various samplers will offer you an assortment of “illustrated” styles, but only certain of them would be what would be classically called True Anime/Manga style.

  • Bad ones:

    • dpm, dpm_a, dpm_fast, uni_*, dpm2, dpm++ STE, dpm++ 2M(usually), dpm++ 2s_a

  • Broken @step=20

    • dpm2_a, dpm adaptive

  • UsuallyGood ones: (But don't use Karras scheduler!) (And they can be inconsistently good. Try different seeds)

    • Euler(mostly), Euler_A (sometimes 3d-ish), Heun. DDIM (sometimes 3d-ish),

    • lms is sometimes usable.

• Hack: change “Anime”, to “Anime, photorealistic” to get more consistent results. An example is the image for this article.
  However… that isnt quite “anime”, is it?

Medium, sometimes large change

(eg: major main character changes. Add/removal of subjects. Change of location)

• Changing base model
  There are some models that can generate somewhat similar results. Others, as mentioned in the prior section, will give you drastically different output

• Changing the Random seed.
  
  
  

• Changing image resolution
  
  For this sample, resolution was changed from 512x512, to 1024x1024. If you go further, and change the aspect ration, it may change the number of people in the image, even if you only give details for a single person in your prompt. Random other details will usually also change.
  
  

• Changing the step count by 10 or more.
  
  Sometimes these don't change much at all. But sometimes they do.
  Important note about tweaking step count: Some base models have a higher minimum than others. SD1.5 models are usually happy at a minimum of 20. However, if you see some things jumbled at that value, start at a higher one.
  Usually good values are 20, or 25.
  For images- larger than 512x512, it is often recommended to use something closer to 35

• Changing the Denoise by 0.1
  
  (denoise 1.0 on left, 0.9 on right)
  This  is sometimes small, but other times jumps to medium change.
  It is possible to change denoise level in both the initial render, and also in an optional refinement/upscale step.  See denoise in next section as well

Typically small change

(eg: some color changes, or increase/decrease of detail in some features)

• Changing Sampler Type
  
  (euler vs dpm++2m)
  Note that some schedulers are sort of in the same family, so you can switch between them with very little change. On the other hand, some others will have much more notable changes. The amount varies with subject matter and Base model)

• Changing Scheduler
  
  
  There is typically only a small amount of change between normal and karras, but occasionally can have larger effects. The first comparison is normal/karras, but the second is normal/ddim_uniform
  
  

• Changing the step count by 1
  
  
  

• Refiner percentage (scale of 0 to 1 )
  Changing 0.1 on a “refine” type operation is sometimes small, but sometimes jumps to medium change

• Denoise value on img2img
  Same effects as “Refiner percentage” (since it’s basically the same type of operation)
  Interestingly, these changes may happen even if you specify the exact same base model, sampler, and scheduler you used in the original image!
  Related to this: If you are upscaling, you may not want to ever have Denoise set to
  > 0.4, otherwise it will change the image too much!