SDXL inpainting works great without CN

You need to merge the "support for SDXL-inpaint model" branch manually as written in this post, or switch to the dev branch of A1111.
Either way, you can use SDXL inpainting model without using the ControlNet Inpaint technique.