Training a Flux Kontext LoRA for Ugly Animal Sketches
Recently, I collaborated with an artist who specializes in quirky, “ugly” animal drawings. Together, we set out to train a Flux Kontext LoRA capable of transforming pet photos into charming, hand-drawn ugly sketches. The results were remarkably creative and consistent, far surpassing our initial expectations. Here, I’ll break down our complete process and offer practical tips so you can experiment with your own Flux Kontext style LoRAs.
The Importance of Paired Datasets
Unlike classic style LoRAs, Kontext LoRA training hinges on paired datasets—meaning each “before” image (the actual pet photo) is matched with an “after” image (the corresponding ugly sketch). This direct pairing is vital because it teaches the model not just a style, but how to translate new images into that style predictably.
Our Paired Dataset
Contents: Real photos—dogs, cats, and animal groups—each matched with a custom ugly sketch by the artist.
Advantages: The artist’s consistent style and creative approach taught the model both the unique look and the transformation logic.
Diversity: Though dominated by mammals with fur, the dataset included numerous breeds and groupings. The model even humorously added fur to dolphins!
Capturing and Captioning the Data
We used BooruDatasetTagManager for captioning and organizing. Our manual captioning approach was simple and direct—each caption described the exact transformation:
Typical captions:
Change the photo of the cat into an ugly sketch of the same cat
Change the photo of the dog into an ugly sketch of the same dog
Change the photo of the animal group into an ugly sketch of the same group
Tips:
Keep captions brief and consistent to help the model focus on the correct transformation.
Avoid over-complicated or vague phrasing, as this can introduce confusion.
Training Considerations for Kontext LoRAs
Key Settings:
Training Duration: Kontext LoRAs need more steps to converge, so don’t hesitate to increase step count or learning rate.
Recommended Workflow
Dataset Preparation: Collect pairs of source and target images.
Captioning: Use tools like BooruDatasetTagManager for manual batch labeling.
Training Configuration: Adjust the training steps and learning rate; use templates that facilitate in-context learning.
Training: Begin training and monitor for signs of overfitting (e.g., the model simply memorizing images).
Results and Observations
Versatility: The Kontext LoRA effectively handled dogs, cats, as well as less common animals and even people in the “ugly sketch” style.
Dataset Limitations: Since our source images were all furry animals, the model tried to add fur even to smooth-skinned creatures like dolphins.
Stylistic Consistency: Thanks to well-paired examples, model outputs closely mirrored the artist’s unique sketches.
If you want to get started, you’ll find an example config file below. With this template and a RunPod instance equipped with an RTX 6000 Ada GPU, training a Kontext LoRA like the ugly animal sketch model should take around six hours.
Example:
Input / Flux Kontext Base / Kontext LoRa
Results:
Kind of a fail, since dolphins don’t actually have fur.
