I have given T5 guidance to CLIP models - mostly focused on SDXL for now.
https://github.com/AbstractEyes/comfy-clip-shunts
Release. It's not idiot-proofed yet but I'll work it out as I go. I still break it occasionally, so if it janks your loadout reload comfy and you should be fine.
The 2048 context window isn't ready yet, so stay tuned for that one soon.
You shouldn't need your HF_KEY but if you do, figure it out. laughs
Apply guidance to your clips by shunting additional information into them from a larger summarization model.
More models will become available in the future as I train them. More adapters will become available as I update the github and train them.
BERT is next on the list, so stay tuned.
Todo:
pony specific trains
illustrious specific trains
omega specific trains
other big models
If you see the CLIP_L or CLIP_G flatline, try some different settings. You killed it with your settings.
Settings:
anchor:
is very potent at shifting concept and keeping similar structure - enable if your prompt isn't taking very well. disable if it's disturbing your prompt too much. You can see the variance in the readout
sigma_log:
increase to add a bit of variance, shift into negative to add negative variance - wild stuff happens if you aren't careful
delta_mean:
your high-heat point, where the natural state is 0 and anything you add will simply amplify the baseline model, anything negative will shift in the opposite internal pool spectrum.
g_pred:
guidance prediction, trained specifically with this model to know cross-conversation guidance potential. it's capped at 2 because it's clamped at 2 internally, it does not go higher.
noise_injection:
injects noise into the embeddings; causing additional disruption and interesting outcomes using standard rand gaussian.
gate_probability:
this determines how willing it will be to close gates when tokens don't conform well enough internally.
How to use:
Git gud.
Nah I'm messing with you. Here's the basics.
Put a T5 loader out like HMYYAHH.
Oh baybe. Yeah don't put a local path for now, it's jank and may not work. Also the sliding window is jank, so just ignore most of the settings and stick the pipe into the conditioners.
It'll just put the flan and the adapters in the HUGGINGFACE_CACHE wherever the hell you have it. I'll put something for that later.
You may be wondering why it only takes one adapter in, but that's okay I made an adapter stacker.
So you can stack as many of them as you want, it'll run them in sequence.
Timesteps work, but it's better if you BACK THEM with the actual prompt.
You'll want 3 prompts; like this.
Hook the merged positive prompts to the T5 loader.
That way you can edit it elsewhere.
Should look something like this.
Fiddle with all the taps and watch the outputs to be sure you aren't just frying everything.
Unguided:
Guided:
Unguided: