Had to enable controlnet inpaint global harmonious preprocessor, lower the strength of this LORA, play with denoising, to keep the image looking and feeling like the same, but the LORA inpainting worked like show after 3 or 4 tries. With enough practice i can feel a perfect harmonious image