Problem

Most guides will tell you to build a diverse dataset, and to be careful to match the style to the target style.

But what if you are making a character lora and

1. you don't have a target style (i.e., you want it to be style independent)

2. you don't have a wide diversity of sources, but only some in a style you don't wish to capture

E.g. your character is a game character - you can get plenty of in game screenshots of them, but now your lora captures the style of the game in addition to the concept of the character.

For example, this lora of Sweety from Zenless Zone Zero which produces the character, but also forces the output to look like an in game screenshot:

Experimental Solution

The solution? Simple - just train a second lora which corresponds to the style of the game, and merge it with the first one at negative strength. The left corresponds to the application of both the positive and negative lora, while the right is only the positive lora, which as you can see still is capturing the look of the game.