TL;DR
Two years of research led to a foundationally powerful series of results, this is one of those results.
https://huggingface.co/AbstractPhil/anima-90k
https://huggingface.co/AbstractPhil/geolip-aleph-diffusion
https://huggingface.co/AbstractPhil/geolip-aleph-differentiation
https://huggingface.co/AbstractPhil/geolip-aleph-qwen
https://huggingface.co/AbstractPhil/geolip-aleph-qwen-3.5-0.8b-instruct
We built a traffic controller for LoRAs. Instead of merging or stacking LoRAs and praying they don't fight, the Aleph MOE-LORA loads several small expert LoRAs under one learned router that decides β per token, per block β how much each expert contributes. Turn an expert off and it's surgically gone: the others don't shift, don't renormalize, don't suddenly change your image. Add a new expert later without retraining the old ones.
To our knowledge, nothing in the current diffusion LoRA ecosystem does learned per-token expert routing. MoE runs the biggest language models on the planet; the LoRA world still combines adapters by static weight math. We've been testing this routing architecture hard on a language-model testbed for months (all of it published, including the failures), the diffusion port is running now, and the early results are why this article exists.
ComfyUI nodes are in preparation alongside the first release: a loader, per-expert toggles with gain dials, and a router inspector that shows you where each expert is firing.
The problem you already know
Every SD power user has lived this:
LoRA soup. Three LoRAs at 0.7 each and the style bleeds into the character, the character bleeds into the pose, and you spend an hour bisecting weights.
Merge and regret. Block-weighted merges are static β the same mix for every token of every prompt, forever.
The removal problem. Drop one LoRA from the stack and everything else changes too, because the combination was never separable in the first place.
These aren't user errors. They're what happens when adapters are combined by addition instead of by decision. Nothing in the stack knows which adapter should own which part of the job.
The structure (what an Aleph MOE-LORA actually is)
Three pieces, all small:
Experts. Ordinary LoRA-sized adapters β each trained on its own slice of the job (a character, a style, a pose family, a lighting regime, structured composition control... your call).
A router at each block. A tiny learned key that reads the hidden state passing through the model and produces one signed weight per expert. Not top-k. No auxiliary balancing losses. Every expert gets a continuous, signed say at every position.
A damped mixer. The weights come from a closed form we call the aleph dispatch:
w_k = sinh(u_k) / Ξ£_j cosh(u_j)
u_k = cos(PΒ·x, C_k) / Ο # P frozen, C_k learned per expert
out = x + Ξ£_k w_k Β· Ξ_k(x) # Ξ_k = expert k's deltaTwo properties of that formula do the heavy lifting:
Signed routing. An expert can push against a direction, not just toward it. Suppression is a first-class citizen.
The denominator never renormalizes. When you disable an expert, its contribution is removed but the denominator keeps its share. The remaining experts do NOT get automatically louder. This is the single biggest difference from every weight-stacking scheme you've used β it's why removal is surgical instead of chaotic.
"Surely someone has done MoE for diffusion LoRAs?"
Mixture-of-experts is standard equipment in frontier language models. In the diffusion adapter ecosystem β merges, stacks, block-weight editing, IP-Adapters, ControlNets β everything combines statically or by fixed schedule. We have not found learned, per-token, per-block, signed expert routing over LoRA deltas anywhere in the current SD tooling. If you know of prior art, genuinely: tell us, we'll cite it. Until then, these models are seeing a structure they have never seen.
Receipts (real numbers, honestly labeled)
We didn't want to sell you vibes, so the routing architecture was pressure-tested for months on a language-model substrate first β same math, same dispatch, published openly with every negative result left in. Highlights from that testbed (these are LLM results; the diffusion port is younger β its data ships as it matures):
Surgical decoupling, demonstrated three ways. Disable the captioning expert and captioning falls to baseline while detection and depth are untouched. Disable detection: only detection dies. Disable the depth expert: the format stays perfect and only the content skill disappears. Three clean double dissociations in one five-expert collective.
Emergent composition. Four specialists that were never trained together produced valid multi-step composite outputs under the router β beating a same-size single adapter trained on everything at once.
The interference bill, measured. Routing isn't free β a five-expert collective costs about what one always-on adapter costs in general-model interference, and roughly 8Γ less than the worst ungated configuration we measured. We publish the whole cost ladder, not just the flattering row.
Toggles behave. Expert masking under the damped mixer is exactly what it claims: all-off reproduces the frozen base model.
Single-seed results are labeled as candidates in our repos; two-seed results are labeled certified. That discipline carries over to everything the diffusion line will publish.
What this unlocks for you
Character + style + pose that don't bleed. Each lives in its own expert; the router learns which tokens belong to whom. Turn the style off tomorrow without touching the character.
Real dials instead of prayer. Per-expert gain on top of learned routing β strength sliders that don't secretly redistribute everything else.
Trainable composition control. Experts trained on structured conditioning slices (layout, subject fixation, depth cues) alongside aesthetic experts β the router decides where structure ends and style begins.
Grow your collection like a library. The expansion path is add-an-expert, freeze-the-rest, brief router refresh. No more re-merging your whole stack because one thing changed.
Inference cost stays sane. Experts are LoRA-sized. The router is a handful of vectors per block. This is not a 4Γ-VRAM proposition β it's a smarter way to spend the adapter budget you already spend.
Training paradigms that actually work
Three recipes, all validated on the testbed:
Specialists first (the cheap one). Train each expert alone on its slice, exactly like training any LoRA today. Freeze them all. Then train only the router β a tiny fraction of the parameters β on the mixed data. The router learns assignment; the experts never fight because they never co-train. This is the recommended entry point and it means your existing per-concept datasets are already MOE-LORA datasets.
Joint mixed-stream (the strong one). All experts + router trained together on interleaved slices, with usage monitoring so no expert starves. More compute, tighter cooperation, and where the emergent-composition behavior showed up.
Add-an-expert (the library one). Take a trained collective, bolt on a fresh expert for a new concept, freeze everything old, short router pass. The old skills stay put β that's the decoupling property doing its job.
The ComfyUI side
Shipping with the first release, as custom nodes:
Aleph MOE-LORA Loader β base checkpoint + N expert files + router weights β patched MODEL out.
Expert Toggle / Gain β per-expert on/off and strength, chainable, honest semantics (damped, never renormalized).
Router Inspector β per-expert usage readout so you can see the router's decisions across steps and blocks.
The node skeleton is done (structure and dispatch math implemented); it goes live with the first trained router release.
Roadmap, and the honest part
Now: SD1.5-family experiments running β including JSON-structured conditioning experts that the base models demonstrably respond to. Next: SDXL and diffusion-transformer (DiT) ports, with the full data drop β training curves, decoupling tables, cost ladders, the works β once the runs are old enough to trust. We publish negatives. If a claim in this article dies in testing, the retraction ships too.
And the honest part: this entire program currently runs on one consumer 4090 and short-term rented pods. The architecture scales β our compute doesn't yet. If the structure excites you, follow the repos, hammer the ComfyUI nodes when they land, and if you're in a position to support serious hardware for the SDXL/DiT phase, that support converts directly into published experiments. Every result β good, bad, and embarrassing β goes public either way. That's the deal.
Aleph MOE-LORA β experts that know their lane, a router that learned the map, and toggles that mean what they say. Data over drama. See you in the release notes.