Can we identify most Stable Diffusion Model issues with just a few circles?

This is my attempt to diagnose Stable Diffusion models using a small and straightforward set of standard tests based on a few prompts. However, every point I bring up is open to discussion.

Each row of images corresponds to a different model, with the same prompt for illustrating a circle.

Stable Diffusion models are black boxes that remain mysterious unless we test them with numerous prompts and settings. I have attempted to create a blueprint for a standard diagnostic method to analyze the model and compare it to other models easily. This test includes 5 prompts and can be expanded or modified to include other tests and concerns.

What the test is assessing?

1. Text encoder problem: overfitting/corruption.

2. Unet problems: overfitting/corruption.

3. Latent noise.

4. Human body integraty.

5. SFW/NSFW bias.

6. Damage to the base model.

Findings:

It appears that a few prompts can effectively diagnose many problems with a model. Future applications may include automating tests during model training to prevent overfitting and corruption. A histogram of samples shifted toward darker colors could indicate Unet overtraining and corruption. The circles test might be employed to detect issues with the text encoder.

Prompts used for testing and how they may indicate problems with a model: (full prompts and settings are attached at the end)

1. Photo of Jennifer Lawrence.

   1. Jennifer Lawrence is a known subject for all SD models (1.3, 1.4, 1.5). A shift in her likeness indicates a shift in the base model.

   2. Can detect body integrity issues.

   3. Darkening of her images indicates overfitting/corruption of Unet.

2. Photo of woman:

   1. Can detect body integrity issues.

   2. NSFW images indicate the model's NSFW bias.

3. Photo of a naked woman.

   1. Can detect body integrity issues.

   2. SFW images indicate the model's SFW bias.

4. City streets.

   1. Chaotic streets indicate latent noise.

5. Illustration of a circle.

   1. Absence of circles, colors, or complex scenes suggests issues with the text encoder.

   2. Irregular patterns, noise, and deformed circles indicate noise in latent space.

Examples of detected problems:

1. The likeness of Jennifer Lawrence is lost, suggesting that the model is heavily overfitted. An example of this can be seen in "Babes_Kissable_Lips_1.safetensors.":

2. Darkening of the image may indicate Unet overfitting. An example of this issue is present in "vintedois_diffusion_v02.safetensors.":

3. NSFW/SFW biases are easily detectable in the generated images.

4. Typically, models generate a single street, but when noise is present, it creates numerous busy and chaotic buildings, example from "analogDiffusion_10.safetensors":

5. Model producing a woman instead of circles and geometric shapes, an example from "sdHeroBimboBondage_1.safetensors". This is likely caused by an overfitted text encoder that pushes every prompt toward a specific subject, like "woman."

6. Deformed circles likely indicate latent noise or strong corruption of the model, as seen in "StudioGhibliV4.ckpt."

Stable Models:

Stable models generally perform better in all tests, producing well-defined and clean circles. An example of this can be seen in "hassanblend1512And_hassanblend1512.safetensors.":

Data:

Tested approximately 120 models. JPG files of ~45MB each might be challenging to view on a slower PC; I recommend downloading and opening with an image viewer capable of handling large images: 1, 2, 3, 4, 5.

Settings:

5 prompts with 7 samples (batch size 7), using AUTOMATIC 1111, with the setting: "Prevent empty spots in grid (when set to autodetect)" - which does not allow grids of an odd number to be folded, keeping all samples from a single model on the same row.

More info:

photo of (Jennifer Lawrence:0.9) beautiful young professional photo high quality highres makeup
Negative prompt: ugly, old, mutation, lowres, low quality, doll, long neck, extra limbs, text, signature, artist name, bad anatomy, poorly drawn, malformed, deformed, blurry, out of focus, noise, dust
Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 10, Size: 512x512, Model hash: 121ec74ddc, Model: Babes_1.1_with_vae, ENSD: 31337, Script: X/Y/Z plot, X Type: Prompt S/R, X Values: "photo of (Jennifer Lawrence:0.9) beautiful young professional photo high quality highres makeup, photo of woman standing full body beautiful young professional photo high quality highres makeup, photo of naked woman sexy beautiful young professional photo high quality highres makeup, photo of city detailed streets roads buildings professional photo high quality highres makeup, minimalism simple illustration vector art style clean single black circle inside white rectangle symmetric shape sharp professional print quality highres high contrast black and white", Y Type: Checkpoint name, Y Values: ""

Contact me.