tldr: Second paragraph
Have you ever used a character lora, either by itself or with a style lora, and noticed it doesn't quite work as expected? As in when it does make the character but only in a single style (the style it was trained with), well, then this article is for you!
Style bleeding mostly happens when the lora is trained for too long and/or on a single style, luckily for us SDXL architecture seems to be robust enough as to (mostly) disentangle such features in some very specific blocks as the images pass through the U-Net, meaning we can get characters with very little to no style bleeding, this also allows for us to get styles without pose/composition/gender bleeding, although this depends a bit on the specific style, taste and what one considers effective stylization, I still haven't done much empirical testing on this, but I will touch briefly on it.
So, how to do it? On forge/automatic11111 you can use this extension (ps: if using multiple loras this extension has a bug so be aware) to be able to selectively enable/disable blocks, here are some reference images of the SDXL architecture and where each block is / what it looks like https://imgur.com/a/K9Rhomv
Now you should be able to use the lora with <lora:my_lora:1:1:lbw=0,0,0,0,0,0,1,1,0,0,0,0>, this will disable all blocks except OUT0 and OUT1, here are some comparison images of how some random loras I picked up from civit look like being used normally <lora:my_lora:1> (all blocks), with lbw=0,0,0,0,0,0,1,1,0,0,0,0 (only OUT0 OUT1) and without it (to show that just those two blocks are actually doing something and the resulting image isn't just the regular model output), all the prompts had "3d" and or "realistic" in it in an attempt to make a 3dfied version of the character
https://imgur.com/a/dH3jUVG
In some examples I had to disable or reduce the OUT1 block weight (natsu (pink hair) and asta (spiked grey hair)) because it was still refusing to properly make a 3D image with it enabled, you will notice that some loras are still able to make 3D versions even with all blocks enabled, but even those can benefit from disabling unnecessary blocks, as the quality of the image improves when they are disabled and the stylization is better applied, one caveat of this is that none of those loras were trained with specific blocks in mind, which means that when we disable or reduce the weight of the blocks the character may become less accurate, there isn't much that can be done about that except enabling back some blocks and seeing the outputs, but if the loras were trained with on those blocks only then maybe the information could've been condensed in them (I'm not really sure as I haven't tested this for training characters)
Now where does all this come from? From reading this and this paper (click on View PDF on the right side of the screen to access the full article), both of them independently came to the same conclusion: in SDXL "style" seems to be mostly present in the OUT1 attention block, with "subjects" mostly in the OUT0 block, and another component of style in the IN08 block ("layout" as per the InstantStyle paper), this seems to be still largely true for pony, but of course it's not all that neatly separated, although the character form is in fact mostly contained in the OUT0 block, a lot of features (especially colors) are still only able to be recovered when enabling the OUT1 block, you can of course also try out the rest of the blocks to see how they affect your lora.
Now how about styles? Well, a similar principle applies to them, you can enable only OUT1 and IN08 to stylize images less aggressively, you may have noticed how some style loras make completely different poses and whatnot from the base model, while this is desirable in some cases (where anatomical differences and other stuff are part of the style) in many others its not, because the style lora ends up interfering with what the base model knows and can do, degrading the quality and variability of the outputs, by training or enabling only specific blocks related to style one can preserve the base model capabilities while still applying stylization, a neat bonus to all of this is that the loras trained on those few blocks are much smaller too.
And what about 1.5? I don't really know, its blocks seem to be a lot more finicky than SDXL, if you want you can try the stuff they mentioned in this paper.
If you want to train specific blocks I recommend using the lycoris config system, an example, I also attached a .txt in this article containing the names of the U-Net blocks to use in the lycoris config.
Loras I used in this: 1, 2, 3, 4
Other articles on this topic:
https://civitai.com/articles/76/bdsqlsz-lora-training-advanced-tutorial1lora-block-training
https://civitai.com/articles/2322/lora-block-weight-extension-in-automatic1111-weights-usage-xyz-plots-for-sdxl-lora
Edit: After training a few loras (both character and style) I can definitely say that you not only can but probably should train only those blocks, in08, out1, or out0, for styles (out1, in8 is optional) and characters/objects/concepts (out0) (specially characters), it will train faster and you will prevent a lot of bleeding and unintended interactions (though some may prefer to train all attention layers for styles so it affects the image as much as possible, you can select the "attn-only" preset on lycoris, also probably conv if you really want to replicate the style as much as possible, keep in mind training all the blocks may re-introduces the risk of overfitting that block-wise training seems to prevent).