V4 img2img support, added image loader so you can start with an existing image instead of ksampler
V3 fixed detection order, now properly detects persons/objects from Left to Right. There is still a manual selection (in green) in case there are more than 2 objects.
V2 added a detailer to fix the faces.
Here is a SD1.5 workflow on how to use multiple character Loras in one image. Essentially it is replacing characters using a Yolo detector/segmenter to perform automatic inpainting, and applying ControlNet to keep the individuals stable in the image. Unfortunately this particular SAM paints the mask using a bounding box, so if characters overlap left-to-right or top-to-bottom strange Lora overlap occurs.
This workflow is designed for 2 people/objects, non-overlapping. The Yolo model I used is designed for semi-realistic initial characters (person_yolov8m-seg.pt from https://huggingface.co/Bingsu/adetailer/blob/main/person_yolov8m-seg.pt). There are other models available, check with Ultralytics or search Huggingface for 'yolo' or 'yolov8'.
The Green boxes are the Inputs.
All 3 KSamplers (and both detailers) are currently set for an LCM checkpoint, so you may need to adjust the steps/cfg if you are not using LCM.
Instructions (top to bottom, left to right):
Load Checkpoint
Load VAE (switcher choses between checkpoint-baked and custom)
Empty Latent Image - set size
Lora Stacker - Set any Loras that you want to apply to the entirety of all 3 images (ex. LCM or Style)
Prompt - Initial image, this goes first and is only used in the initial image
Prompt - Background append, this goes last and is applied to all 3 images
Prompt - Person A, this generally describes person A (may be altered later)
Prompt - Person B, this generally describes person B (may be altered later)
Seed - Leave at fixed, adjust this when needed
Select Person - Detects left-to-right if 'Select Person A' is 0. Only need to select if there are more than two, as of Version 3. Queue prompt once and see if the White area corresponds to Person A (set take_start to 0, 1, 2, etc.) If it is not correct, keep the Seed the same change take_start and Queue Prompt again.
String Replace - Replace the generic description of Person A with a Lora trigger if needed (example: in old type "a girl", in new type "mona_lisa_lora_trigger" or whatever)
Load Lora - Person A Lora
Remaining Person - There is a manual selection box now, modified in Version 3. Leave take_start at 0 unless there are extra people detected, then adjust as necessary (the White area is the person detected)
String Replace - Replace the generic description of Person B with a Lora trigger if needed
Load Lora - Person B Lora
(First run, set yolo model in the lower left-section)
Queue Prompt
You may want to play with the denoising values depending on situation. I found that 0.8-0.9 works well for me. Your mileage may vary.
Facedetailer could be added after the KSamplers for Persons A and B for enhancement. This may get added in a future version. Added in V2.
Could be modified to use XL models without too much fuss. Just need to change KSamplers and add a refiner.
Files used in the workflow:
ControlNet Openpose 1.5 - https://huggingface.co/lllyasviel/sd-controlnet-openpose/blob/main/diffusion_pytorch_model.safetensors
Yolo person model - https://huggingface.co/Bingsu/adetailer/blob/main/person_yolov8m-seg.pt
Yolo face model - https://huggingface.co/Bingsu/adetailer/blob/main/face_yolov8m.pt
Let me know your comments below!