How to merge SDXL clips into SD3?

Here is a experimental test model I have created for testing and science usage.. https://civitai.com/models/511463/afros-sd3-experimental-sdxl-merge

Many asked me how its possible to combine SDXL clips with SD3 ones.

First you have to know that SD3 can uses typically three text encoders. The first two of them are Clip-G and Clip-L. Those are basically the same as the ones from SDXL (correct me if I am wrong).

The third one is the new T5 text encoder. This text encoder can't be merged with the SDXL ones..

Here is the detailed research paper published by Stability AI https://arxiv.org/pdf/2403.03206

Note that you only merging the text encoders here not the whole checkpoint!

To start download this model here. https://civitai.com/models/497255?modelVersionId=568278

This is the SD3 model that includes the two text encoder I mentioned above (clip-L and clip-G).

After that start Comfy AI.. You need some costom nodes here from ComfyRolls (created by Suzie1)

You can install them with the Comfy Manager or just download it from here: https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes

The process itself is pretty easy..

MERGING SDXL LORAS into the SD3_clip.

The model weight should not have an impact.. Only the clip ratio. I don't want to reveal my exact ratios here. You can try any Lora you want.. But it won't maybe not have the effect you are wanted because this only affects the clips.. In my case I used a own trained SDXL enhancer. For me it increased the details, sharpness. not the image itself.

Merging SDXL models into the SD3-clip

You can also merge other SDXL Clips with SD3 models.. In my case I used a own SDXL model very similar to my Colossus Project 10B checkpoint. https://civitai.com/models/147720?modelVersionId=557262

The result was interesing.. It actually changed the image this time. The model got more coherent..

after

Now its time to test all resaults.. I always create different ratios and test then all models against each other. For the example experimental model I merged the best models a second time.

This is what I got as final result: (example)