UPDATE : I included a special inpaint to add details and also a new method to re-imagine an input image (without control net or ip-adaptors). Also a sdxl only workflow and one with double control nets. Here i'm using the 8bit T5 from sd3. You can change that with a bigger model fp16 or full precision if you want.
This workflow does 15 steps (or 20) with pixart sigma and following with 15 steps (or 20) on sdxl. Sometimes i like to keep more of the base image and i run 10-12 steps in sdxl. Just play with the values.
Keep the loras of the sdxl model there, helps with anatomy. Playing with cfg for 4-8 for both models gives the best results.
Do not mistake this workflow with an upscaling one, this is not an upscale, this harmonizes both models by transferring latents and getting a coherent image without upscaling artefacts so you get the best from both worlds. You get excellent composition from Pix-art Sigma and then get to have excellent quality, colors, style, anatomy with the sdxl pass. Make sure to use a sdxl model with great anatomy and prompt comprehension. So you can use my sdxl model or bake DPO (direct preference optimization) into your favorite models. Not sure Pony counts as a model with good anatomy because it tries it's best to copy source art, and does not have good anatomy without doing so, but try and see, maybe i'm wrong.
Installation Instructions :
Install the latest torch and xformers in comfyui (recommended).
pip install torch torchvision torchaudio xformers --extra-index-url https://download.pytorch.org/whl/cu121
It is better to clone the comfyui repo in a folder, then install everything in a conda environment. That said if you don't want to use the comfyui portable edition and want to install xformers. This was the way i did it.
conda create --name comfyui
conda activate comfyui
Install extra models from comfyui manager (updating the node necessary)
Download the Sigma model from : Here
PixArt-Sigma-XL-2-1024-MS.pth
Download the following T5 and rename them as seen : Here
Or if you wish to use the 8bit T5 download it from stability sd3 medium hf repo.
config.json
model.safetensors.index.json
model-00001-of-00002.safetensors
model-00002-of-00002.safetensors
t5xxl_fp8_e4m3fn.safetensors
UPDATE : you can use only the 8-bit T5 if you wish (4gb).
Download this VAE file from : Here
Rename it to pixart_sigma_vae.safetensors
Update : The normal sdxl vae works too.
Place them inside the ComfyUI models directory as follows :
comfyui/models/
/checkpoints
> PixArt-Sigma-XL-2-1024-MS.pth
> ReyMix_XL.safetensors (or any other SDXL model)
/t5
> config.json
> model.safetensors.index.json
> model-00001-of-00002.safetensors
> model-00002-of-00002.safetensors
/clips
> t5xxl_fp8_e4m3fn.safetensors
/vae
> pixart_sigma_vae.safetensors