Finetuned LoRA for Enhanced Skin Realism in Qwen-Image-Edit-2509
This repository contains a finetuned Low-Rank Adaptation (LoRA) model designed to enhance the realism and detail of human skin in images. The LoRA has been trained on top of the powerful Qwen/Qwen-Image-Edit-2509 model, leveraging its advanced image editing capabilities to focus specifically on generating more natural and detailed skin textures.
This model was trained for 5000 steps on a local RTX 5090 using the AI-Toolkit. The resulting LoRA is ideal for photographers, digital artists, and anyone looking to improve the quality of human subjects in their generated or edited images.
Model Description
The qwen-edit-skin LoRA is a specialized finetuning of the Qwen/Qwen-Image-Edit-2509 base model. The base model is a versatile image editor with strong capabilities in multi-image editing and maintaining single-image consistency, particularly in preserving personal identity. This LoRA builds upon that foundation to specifically address the nuances of human skin, adding detail and realism that may not be present in the original generations.
The training was conducted using this fork of AI ToolKit, a comprehensive suite for finetuning diffusion models. The process for curating the dataset involved reverse modification of subject skin details as follows:
Taking real images of versatile subject portraits with skin exposed
Captioning each of these as our “Target” (THE AFTER) images for the final outcome expected in a standard Qwen Edit workflow
Editing the image in Photoshop to add more gaussian blur and smoother skin tones, to make the skin texture, tone and pores less visible
These became our “Control” (The BEFORE) images for Qwen Edit training.
Training Details
The model was finetuned with the following key parameters, which can be found in the accompanying config.yaml file:
Hardware:
GPU: NVIDIA RTX 5090
Training Configuration:
Training Steps: 5000
Batch Size: 1
Gradient Accumulation: 1
Learning Rate: 1.0e-04
Optimizer: adamw8bit
Noise Scheduler: flowmatch
Resolution: The model was trained on a dataset with resolutions of 512, 768, and 1024 pixels.
Precision: bf16
Network Architecture:
Type: LoRA
Linear Rank & Alpha: 16
Convolutional Rank & Alpha: 16
The choice of adamw8bit as the optimizer is significant as it reduces the memory footprint of the training process, allowing for more efficient finetuning on consumer-grade hardware without sacrificing performance. The flowmatch noise scheduler is a modern approach that can lead to more efficient training and high-quality image generation.
A notable aspect of the LoRA architecture is that the alpha values for both linear and convolutional layers are set to be equal to their respective rank (16). This balanced approach is a common starting point for LoRA training, ensuring that the learned adaptations are applied with a proportional scaling factor, which can help in preventing overfitting while allowing the model to learn the desired new features effectively.
How to Use
To use this LoRA, you will need to load the base model Qwen/Qwen-Image-Edit-2509 and then apply the finetuned LoRA weights loaded as qwen-edit-skin.safetensors. Previous step versions of the weights are uploaded for reference but the final version is qwen-edit-skin.safetensors. You can also leverage the example workflow attached in the repo for ComfyUI to compare the results across different weights.
The recommended weight is between 1 and 1.5, the examples provided show weights up to 2 only to show the effect of the Lora with a strength considered too high for effect.
Intended Use
This LoRA is intended for creative and artistic purposes to enhance the realism of human skin in digital images. It can be used by:
Digital Artists: To add finer details and textures to the skin of their characters.
Photographers: For retouching and enhancing portraits.
AI Art Enthusiasts: To generate more lifelike images of people.
Limitations and Bias
This model is a finetuning of a large-scale, pre-trained model and may carry some of its inherent biases. The training dataset for this LoRA was focused on improving skin details and may not represent the full diversity of human skin tones and types equally. Users should be aware of this and use the model responsibly. The output of the model is influenced by the input prompt, and users are encouraged to use descriptive and inclusive language to guide the generation process.
Disclaimer: This model is intended for artistic and creative purposes. Users are responsible for the content they create and should adhere to ethical guidelines and respect the privacy and dignity of individuals.
Trigger words
You should use make the subjects skin details more prominent and natural to trigger the image generation.