Introduction
This is something that I (and probably many others) have noticed that certain custom models tend to produce faces that look similar to each other (and I'm not talking about models tuned on to produce a single face like Emma Watson's.) Often, I can guess, by the AI image itself, what model is used to create it (or a mix containing a sizable portion of that model.) Custom models are great for producing higher quality and more specialized images but one of the side effects of this is that the people's faces, while being of higher quality, are much less diverse compared to the base SD 1.5. I'll explore some sample generations with different models and different ethnicities of women's faces (focusing on women here since I'm guessing most of us here are far more interested in them compared to men..)
To start, let's generate 10 images from 10 different SD1.5-based models each, making 100 faces total. All images were generated are 512x512 using Euler A as the sampler and without high-res fix. Note that most of the models I'm using are for photorealistic pictures and may not be the latest versions. The VAE I'm using is the standard vae-ft-mse-840000-ema-pruned (even if some of the models have their own VAEs embedded since others might not have a VAE and I have to select a uniform VAE option among all models when generating a grid.) Also keep in mind this is an informal study of the issue, not an Arxiv paper so I'll be using a lot of informal terms. If anyone knows of a more rigorous metric for measuring facial diversity, please let me know and I may do a follow up study.
The following prompt is used:
Prompt: a woman's headshot, looking at viewer
Negative Prompt: full_body, b&w
Note that "headshot" helps create a professional looking photo with the entire face in the picture and in focus (compared to just "face" or "head".) Putting full_body in the negative prompt helps leave out the body.
Here are the results:
And here are some of my observations based on the different models:
SD1.5: A decent blend of lighter and darker skinned women, probably typical of the demographics in the US.
Absolute Reality 1.81: Looks like they're all of the same person
Beautiful Realistic Asians 5: As the name suggests, they're all Asians but it seems only 2 or 3 individuals are represented in the 10 images generated.
Beautiful Realistic Women 1.0: It's created by the same author as Beautiful Realistic Asians but intended for all races but it seems the faces have an Asian characteristic to them, possibly Asian-White mixed race women. And there's not as much facial variety.
Epic Photogasm Last Unicorn: Faces all look similar, there might be 2 different people but even then, they look related and their facial structure all looks the same.
Epic Realism Natural Sin RC1 VAE: Same creator as Epic Photogasm and has even more similar looking faces. This model was originally recommended by a Youtuber who claimed it was even more photorealistic than SDXL and while I was impressed by the photo quality, I felt the facial variety was lackluster which inspired me to eventually do this study.
Humans v1.0: This model was specifically trained to produce more "normal" looking faces with imperfections than ideal faces that most of the other models have. Not surprisingly, the facial variety seems to match SD1.5 and I suspect it was fine tuned directly from that since I seem some similarity between these faces and the faces from SD1.5 using the same seeds. This also seems to be the only other model that generates darker skinned women when you prompt for a generic "woman".
Objective Reality 2.0: Same creator as Humans 1.0, this photorealistic model focuses on quantity rather than quality of concepts. It's been my go-to model for fine tuning and the creator suggests you use it to fine tune since it's more of a jack of all trades model. Facial variety-wise, it seems to only generate Caucasians though with greater variety than some of the other photorealistic models feature here.
Photon V1: an average variety of faces, similar to Objective Reality 2.0
Realistic Vision 5.1: Not only do the faces look quite identical to each other but they don't seem to differ much compared to those from Epic Photogasm for the same respective seeds. Still, this is a very high quality photorealistic model from my experience but just don't expect it to generate a large variety of faces.
How do these models fare when generating women of different races?
A lot of these models clearly exhibit racial bias as well as low facial variety when you simply prompt for "woman" but what if you specify the race in the prompt. Let's find out:
Asian Women
Prompt: an Asian woman's headshot, looking at viewer
African Women
Prompt: an African woman's headshot, looking at viewer
Hispanic Women
Prompt: a Hispanic woman's headshot, looking at viewer
Caucasian Women
Prompt: an European woman's headshot, looking at viewer
Indian Women
Prompt: an Indian woman's headshot, looking at viewer
Middle Eastern Women
Prompt: a Middle Eastern woman's headshot, looking at viewer
As expected, these models are perfectly capable of generating women of a variety of different races if you specifically prompt for them although the same models still seem to suffer from a lack of facial variety, just amongst women of the race you've prompted.
Ways of increasing facial variety
This is something I'm still experimenting with and I've had mixed results with. I'll share what I've tried (though I may not have time to show example images from all of these) and you can feel free to share what you've tried in the comments as well.
Using common first names
Add common first names to your prompt like "Sarah","Jane","Lauren", etc. Some people will put these names in a Dynamic Prompts wildcard and have the prompt select a random name from the list a random. This seems to be the easiest way to increase facial variety though the effects seem a bit limited. It's helpful if, for some idiosyncratic reason, you're constantly getting very similar looking faces for a particular prompt that you're using even though the faces may look different for other prompts.
Using LoRAs
And I don't mean just loading an Emma Watson or Taylor Swift LoRA and calling it a day unless you want to generate countless pictures of Emma Watson or Taylor Swift which basically defeats the purpose of facial variety. You can also download a ton of LoRAs and put their triggers into a Dynamic Prompt file (each with something like <lora:loraname:0.5> though you may need to replace the 0.5 with the optimal weight for that LoRA.) At least this way, you'll get variety though your hard drive can be cluttered with countless face LoRAs. To get the most out of it, you can further increase the # of possible faces using mixing in your prompt:
[ __faces__ : __faces__ ]
Assuming you're using a file called "faces.txt" for your dynamic prompts and you have a face LoRA on each line, this will pick a random face from one line of your file, and another face from another random line of your file (possibly but most likely not the same as the first one) and mix them together, squeezing out far more facial variety from your collection of LoRAs.
You can also use LoRAs that have multiple faces trained into them (like my Asian Doll Likeness and Eastern European Doll Likeness LoRAs), each with their separate trigger word then put all of the trigger words in a Dynamic Prompt and pick one from random or do the mixing trick above for even greater facial variety. While LoRAs with multiple faces in them might not produce each individual face as well as a LoRA dedicated to a single person's likeness, it may serve the side purpose of simply increasing facial variety.
One last method I've considered is to train a LoRA on thousands of different faces and without labels but with a single trigger word. This may be less tedious than any of the methods mentioned above and is starting to show promise of increasing facial variety if you train the LoRA with SD1.5 (or maybe Humans) as the base. I'll let you know in future postings and/or by releasing the LoRA once it's ready.
Final Thoughts
After seeing people posting pictures of AI generated women with similar looking generic faces, I thought I might offer a few tips here to hopefully increase their variety. When people start to associate the same style of face with the art being "AI generated", the lack of facial variety amongst popular models has become quite apparent. While I'm far from an expert in this subject, I'm hoping people will at least try some of these tips to create a wider variety of faces among the pictures of AI women that we generate.