Introduction
This is a list of the extensions that I am currently using for my inpainting workflow. Some of these are not directly related to the inpainting process but are helpful towards the brainstorming process. For the scope of this guide, I will be trying to avoid mentioning paid resources. If I had to recommend using money for something, I would recommend getting a tablet first since it's quite the force multiplier. I will be laying out the sections as use cases since the future is very uncertain and new tools could end up replacing older ones. This guide assumes that you using the A1111 webui.
Must-Have
Freely Edit with an Image Editor
For the inpainting process, it is vital to have the flexibility control your mask size along with the ability to make minor edits to the image. Inpainting isn't a solution to everything and occasionally small edits are faster than another inpainting run. Having integration with an image editor is the best quality of life enhancement for inpainting.
Krita
https://github.com/DrCyanide/cyanic-sd-krita (A1111)
https://github.com/Acly/krita-ai-diffusion/ (ComfyUI)
I personally have use cases for both plugins where each plugin has their own strengths and weaknesses. Krita AI Diffusion aims to have a more simplified UI for inpainting while the options with Cyanic SD Krita is more similar to A1111.
Photopea
https://github.com/yankooliveira/sd-webui-photopea-embed
This extension integrates the photopea webapp directly into Stable Diffusion webui allowing you to make edits and directly send the results to txt2img,img2img or ControlNet. Offers utilities similar to photoshop but cannot be run offline. Photopea has better editing capabilities compared to Krita but I just like the comfort that Krita provides without needing to adjust my tablet.
Additional Model Controls
One of the general limitations with generative AI is that it will often struggle to create our desired image with just prompts alone so more ways to add/remove/manipulate the model is always welcome. One of the more popular ways to do is by using LoRA models which can inject missing data into the checkpoint and guide the result, but making a LoRA for everything can be time-consuming and we're not entirely sure if every LoRA can be flexible enough for all use cases. Thus one of the biggest focuses on generative AI is trying to find ways to give additional instructions to the checkpoint.
ControlNet
https://github.com/Mikubill/sd-webui-controlnet
This extension enables the usage of ControlNet type models which can apply conditions to a checkpoint's output. It is one of the biggest game changes for Stable Diffusion and the current cutting edge technology for manipulating models. The extension comes with support for lineart ,color, depthmap, normal map, and many other options for output control. While training for custom ControlNet models isn't common, they do exist and is something you should be on lookout for more options. ControlNet itself is a fairly complex topic so I won't be going into more detail about it here.
Lycoris
https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris
Allows you to use the Lycoris extended LoRA type. Serves the same purpose as LoRAs for adding missing data but support for Lycoris is not built-in with the webui by default. Occasionally, you might find a Lycoris with some detail you want so have a broader scope for more data isn't a bad idea.
LoRA Block Weight
https://github.com/hako-mikan/sd-webui-lora-block-weight
This extension allows you to manipulate the block weights in LoRA models which can help reduce overfitting and concept bleed. A block weight is a component in a model that manages the generative output over shapes and colors. Being able to control a block weight gives you more control over a LoRA and can enable LoRAs to work better with checkpoints of differing styles.
Regional Prompt Control
While these types of extensions are not a complete must have for every workflow, they are helpful towards maintaining specific prompt control over regions of an image. If you have an exact idea of what your output will look like, then this extension can help greatly.
These extensions are more valuable in multi-subject workflows where the prompt matters a lot more compared to single subject generation. Multi-subject workflows tend to suffer from concept bleeding which may cause unintended results of character details blending together. To prevent this issue, these extensions are considered a must-have for that particular use case. These extensions are not perfect and generations could still suffer concept bleed issues but regional control does help make the process more controllable.
There's quite of number of extensions that handle this particular use case so I'll just mention the two most common ones. There's a bit of a learning curve to these tools so https://stable-diffusion-art.com/regional-prompter/ can help out.
Regional Prompter
https://github.com/hako-mikan/sd-webui-regional-prompter
One of many tools that allow you to specify a region by mask and then apply a specific prompt to that area.
Latent Couple + Composable LoRA
https://github.com/ashen-sensored/stable-diffusion-webui-two-shot
https://github.com/a2569875/stable-diffusion-webui-composable-lora
Note: You can only use Regional Prompter or (Latent Couple + Composable LoRA), you cannot mix these extensions together.
Latent Couple (TwoShot) and Composable LoRA have the same functionality as regional prompter so it's basically preference as to which one is easier to use.
The main advantage of latent couple + Composable LoRA is that is combo handles concept separation of LoRAs better compared to Regional Prompter. However, it is not perfect.
Cutoff
https://github.com/hnmr293/sd-webui-cutoff
An extension that helps with preventing color concept bleeding. It's not perfect but can be helpful. It may require tinkering with the weight.
Outpainting
While it is tempting to generate widescreen images or large images with Stable Diffusion at once, Stable Diffusion isn't able to generate high resolutions without errors and you will need to resort to outpainting or upscaling methods to create these images. Unlike upscaling, outpainting increases the size of the image by expanding the canvas, giving more flexibility of adding more detail to the image.
ControlNet
https://github.com/Mikubill/sd-webui-controlnet
This extension was mentioned earlier but recently, ControlNet received a big update that allows outpainting using the inpaint only processor which effortlessly defeats all the other older forms of outpainting. The only downside is that ControlNet-based generative fill uses another hi-res fix pass on the backend so it's much slower compared to the older outpainting method but it's far less error prone.
Upscaling Tools
Increasing the image resolution while maintaining image quality is fairly common use case. Stable Diffusion comes with support for different upscaling methods which differ on the artstyle. Upscaling tools aren't perfect and may add changes to image or could end up adding extra details to enhance it. You can find upscalers on the https://upscale.wiki/wiki/Model_Database upscale wiki.
MultiDiffusion
https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111
MultiDiffusion is an extension that uses a tiling algorithm to upscale images. This comes with the benefit of being able to upscale large images with a lower amount of VRAM. The downside is that it does slow down generation times. I personally find this extension easier to use compared to SD Ultimate Upscaler despite all of the additional switches.
SD Ultimate Upscaler
https://github.com/Coyote-A/ultimate-upscale-for-automatic1111
This is a simple script that allows you to do tiled based upscaling similar to MultiDiffusion. Between the two extensions, I found that SD Ultimate Upscalers tends to have its final output closer to the original but I have found it harder to configure without ending into errors.
Model Organization
After downloading a ton of checkpoints and LoRAs, it's very difficult to tell apart which file is which due to similar naming. It's a lifesaver to have a tool that can track triggerwords and have some better form of model management.
SD Webui Civitai Helper
https://civitai.com/models/16768/civitai-helper-sd-webui-civitai-extension
A helpful extension that can autofill your model previews, detect model updates(not automatic), and stores triggerwords.
Quality of Life Improvements
These extensions are not necessary but give some nice quality of improvements.
Prompting Tools
A set of tools that helpful towards prompt engineering.
TagComplete
https://github.com/DominikDoom/a1111-sd-webui-tagcomplete
Enables Booru style tag auto completes in the webui. It's an amazing time saver.
Dynamic Prompts
https://github.com/adieyal/sd-dynamic-prompts
Allows wildcard and combination testing of prompts.
Background Remover
Fairly self-exploratory. Tools that can used to remove a background.
PBRemTools
https://github.com/mattyamonaca/PBRemTools
A background remover tool based on SAM.
Inpainting Support Tools
A set of tools that can help with inpainting but are not strictly necessary.
Canvas Zoom
https://github.com/richrobber2/canvas-zoom
Enables the ability to zoom in on images for inpainting. You don't need to download this extension since it will be directly integrated into the a1111 webui for the 1.4 update. Pointing it for the people who are using older versions for more stability.
ADetailer
https://github.com/Bing-su/adetailer
This tool allows automation of inpainting for faces and bodies. It doesn't use a precise mask over the subjects but can be helpful for increasing the details of the image if you have a ton of people in the background.
Honorable Mention
Personally, I haven't been able to get these extensions to work but I thought that they could be useful for somebody since they introduce an interesting concept.
Inpaint Anything
https://github.com/Uminosachi/sd-webui-inpaint-anything
A tool that uses SAM to automate the creation of precise inpainting masks. This extension can be helpful in some use cases but SAM isn't 100% accurate. This is the better tool but there is a lack of integration with MultiDiffusion so you'll end up ending into a lot of out of memory errors.
-edit- This tool now has support for SAM HQ which is more accurate and captures more subareas.
Segment Anything
https://github.com/continue-revolution/sd-webui-segment-anything
Another tool that uses SAM to create inpainting masks but allows you specify which regions to divide. It comes with multiple features such as object bounding box detection, and batch processing. It's a fairly complex tool that I haven't fully understood yet.
Krita Segmentation Tools
https://github.com/Acly/krita-ai-tools
A plugin for Krita which adds smart object-like selection for Krita. It can be error prone but is a nice quality of life improvement for Krita.
Automatic Layer Division
These type of tools are still in early in development and are very imperfect but I see a lot of potential with being able to help out with general editing.
Stable Diffusion webui- Layer Divider
https://github.com/jhj0517/stable-diffusion-webui-Layer-Divider
A tool that uses SAM to divide the image into layers and saves as a PSD file.
Layer Divider
https://github.com/mattyamonaca/layerdivider
Another layer division tool that uses SAM but is able to subdivide the layers into groups of shadow, and lighting.
Posing Tools
A set of tools that can be used together in conjunction with ControlNet.
openpose editor
https://github.com/fkunn1326/openpose-editor
A simple 2d editor for adjusting poses.
sd-webui-openpose editor
https://github.com/huchenlei/sd-webui-openpose-editor
A simple 2d editor for adjusting hands and pose within the controlnet ui. Has the ability to hide the visibility of selected bones.
3d open-pose-editor
https://github.com/nonnonstop/sd-webui-3d-open-pose-editor
A 3d pose editor that is integrated into Stable Diffusion.
SD-webui-depth-lib
https://github.com/jexom/sd-webui-depth-lib
A 2d editor that allows you to adjust the position of depths for hands. Helpful for fixing hands with ControlNet.
Posing Resources
External resources for posing.
Blender Character Bones
https://toyxyz.gumroad.com/l/ciojz
A blender model with posing utilities with ControlNet
MagicPoser
https://webapp.magicposer.com/
A generic model posing website with a lot of presets for adjusting the model
Posemyart
A generic model posing website that comes with support for exporting pose data for controlnet.
Posemaniacs
https://www.posemaniacs.com/en/tools/viewer/hand
A site that contains a viewer with presets for 3D models of a hand that can be freely rotated or resized. This can be a helpful tool with fixing hands with ControlNet. Unfortunately, the hand cannot be adjusted and additionally there is no easy to download the image so you will need to take a screenshot.
Prompt Image Viewers
Utilities that let you see the prompt connected to the image.
BreadBoard
https://github.com/cocktailpeanut/breadboard
This application doesn't seem to be updated as often any more but it allows you view your prompts without having to start up Stable Diffusion
Image Browser
https://github.com/yfszzx/stable-diffusion-webui-images-browser
An extension that integrates an image browser into Stable Diffusion
Other
Latent Couple Region Mapper
https://badnoise.net/latentcoupleregionmapper/
A web-app that helps with mapping regions and crafting prompts in combination with 'latent couple' or regional control extensions.
AI Tag Library
A very simple image gallery with comparison of keyword effects. Anime-focused.
Optimizations
Stable Diffusion webui-TensorRT
https://github.com/AUTOMATIC1111/stable-diffusion-webui-tensorrt
Enables the conversion and usage of tensorRT models. These models are faster than the standard model but have image size generation limits. Luckily, tensorRT seems to be able to support 768 x 768 dimensions so you can use tensorRT models for more speed during a 'mask only' inpaint quality pass. Do note that it does take some technical knowledge to get this extension working with the webui.
End
Thanks for reading and maybe this was helpful for somebody!
Changelog
-fixed errors with the inpainting anything extension. I still don't know enough about them to draw a good conclusion
-6/24/23 Added the latent couple region mapper, taglib
-6/28/23 Forgot to add "SD Webui Civitai Helper" and model management use case
-8/11/23 Updated section on regional control and added posemaniacsas a resource
-12/14/23 Updated recommended plugins for Krita (a1111/comfyui/segmentation tools)