Training LoRAs for Realistic People on Civitai with Pony Model Stable Diffusion

In the world of AI-generated art, Stable Diffusion models have revolutionized the way we create realistic depictions of people and fictional characters. However, achieving a high degree of accuracy in portraying real individuals has often been a challenge. Many LoRAs (Low-Rank Adaptations) of realistic people created for pony models uploaded to platforms like Civitai tend to miss the mark in resembling the actual person.

Having faced similar issues, I ventured into training LoRAs myself, focusing on the Pony model mora recently rather then SDXL which I used to do more. Within Stable Diffusion, Pony offers more control over the subject, which is crucial for creating realistic portrayals of fictional characters my main goal. By combining different LoRAs, like using a model for Sasha Zotova (the face model of Jill Valentine) with Jill Valentine's outfit which pony already knows, one can achieve remarkably accurate and lifelike images.

In this article, I'll share my findings and provide a step-by-step guide to help you train your own LoRAs for realistic people using the Pony model on Civitai. These techniques have significantly improved the accuracy of my models, making them closely resemble the intended person.

Example of a LoRA for Stefanie Joosten

Here is an example of my LoRA for Stefanie Joosten that I trained yesterday. Note that the first few images are raw epoch examples of different epochs.

The final image shows what you can do with these kinds of LoRAs, as I have made an example of Stefanie as Supergirl.

This, of course, isn't my goal with this LoRA. As you can probably guess, I aim to create her as Quiet, but I am still working on that.

Step-by-Step Guide to Training LoRAs on Civitai

1. Setting Up Your Training Environment

First, you'll need to set up your training environment on Civitai. Here are the basic settings and adjustments you'll need to make:

- Base Model: Start with a realistic Pony model. Two excellent options are Zonkey Realism and Valiant Stallion.

- Training Data: Collect a diverse set of images of the person you want to train the LoRA for. The quality and diversity of your training data are crucial.

2. Adjusting Training Parameters

To achieve better results, modify the following settings in Civitai:

- UNet Learning Rate: Set this to 0.00100. This learning rate tends to work well with realistic Pony models.

- Text Encoder Learning Rate: Leave this as is.

- Epochs: Increase the number of epochs to 20 instead of the default 10. More epochs allow the model to learn better from the training data.

3. Captioning Your Data

- Auto-Captioning: Let Civitai handle the auto-captioning. This feature helps to automatically generate descriptions for your images.

- Manual Captioning: The only manual caption you need to add is the name of the specific person. This helps the model to associate the correct identity with the images.

4. Optimizing the Learning Process

- Learning Method: Set the learning method to 'cosine'. This method helps in achieving a smoother convergence during training.

- Optimizer: Use 'prodigy' as your optimizer. This setting is effective for fine-tuning models to achieve better accuracy.

5. Evaluating the Results

After training, evaluate the results of your LoRA. Typically, 8 out of 10 times, one of the 20 epochs will yield a near-perfect result.

Final Thoughts

I am not a professional, but I hope these tips help you in training Pony models for realistic people. Here are a few key points to remember:

- Quality training data is essential.

- Adjust the learning rate and epochs for better results.

- Use a combination of auto-captioning and manual captioning.

- Respect the individuals you are creating models of and avoid any misuse, such as creating deep fakes.

Additionally, I will not be uploading my Stefanie Joosten LoRA. I have noticed that more and more people misuse Stable Diffusion to create deep fakes of real persons, which I strongly oppose. In fact it is for this reason I have unpublished al my LoRA's of real people.

I hope that if you use this knowledge, it will be to create content that is respectful to the person.

By following these steps, you should be able to train LoRAs that closely resemble real people, allowing you to create accurate and respectful content.

Happy training!