ControlNet IP adapter can be kind of a finicky beast. At its default settings, it will produce an oversaturated image that will completely subsume your prompt. When set up correctly, however, it can impart some amazing styles to your images that wouldn't be feasible with word prompts alone. As always, I want to stress that no one particular set of settings is the definitive way to do it -- different settings can produce different effects, so you should use my example here as a baseline and experiment from there.
ControlNet Settings
Note that these settings are for SDXL. Optimal settings for SD1.5 may differ significantly.
Preprocessor and Model - make sure that these are both set correctly depending on whether you're using SD1.5 or SDXL.
Control Weight - 1.0 has been too high in my experience, and tends to result in overcooked images. I've had better luck in the 0.3-0.6 range.
Starting Control Step - The higher you set this, the more your prompt will influence the composition of the image. I've had good luck between 0.1 and 0.3. Leave this at 0 and you won't see much of your prompt at all most of the time.
Ending Control Step - I've left this at 1, but you might get some interesting effects if you experiment with it!
Subject matter bleed
The subject matter of your IP Adapter image will usually bleed into your generations (If your style image is a Japanese nature watercolor, your output will probably have Japanese shrine and cherry blossom trees, etc.). This can be really interested in terms of adding flavor to your outputs, but you may have a particular genre of output in mind, in which case you might not want this to happen. If you're having trouble with this, use an image that's a similar genre to what you're going for, or use a completely abstract image. Or, just accept that the subject matter bleed will happen and enjoy the results!
Where do I find good style images?
Generate them yourself. Use Stable Diffusion, Midjourney, Bing Image Creator, and ChatGPT+DALL-E 3. That swirling rainbow oil paint image in my settings was from DALL-E 3.
Make them yourself. Use photoshop, krita, blender, photos of paper cuttings, or anything else you can think of.
AI art sites. A lot of the featured images right here on CivitAI are in very interesting styles, which makes them excellent for this. Midjourney's showcase page also has a lot of great stuff, but you need a special browser extension to download it all (this is outside the scope of this article, but just google it).
Art sites like DeviantArt, Pinterest, Artstation, etc.
WikiMedia Commons is a great place to grab stuff by the Old Masters.
Experiment!
As usual, if you only take one point from my articles, it's to experiment with different things and see what you get. If you come up with anything interesting, please link it here, because I'd love to see it!
