Sign In
homemodelsimagesvideos3D Modelshubsarticlescomicscruciblesupdatesshop

How I Made the Qwen3.5 Text Encoder for Anima (and Can You Use a Bigger LLM?)

0

kayfahaarukku's Avatar

kayfahaarukku

Sleepless Nights

Oct 17, 2026

training guide
How I Made the Qwen3.5 Text Encoder for Anima (and Can You Use a Bigger LLM?)

A few of you asked how I made the Qwen3.5-0.8B Text Encoder for Anima, and one person asked if the same thing works on a bigger LLM. So here's the whole recipe.

The good news: it's cheap. The GPU part took about half an hour on my aging RTX 2060 12GB, and it needs no images and no Anima training at all. The answer to the bigger-LLM question is at the end, and it's not "no". It's "yes, but this recipe won't give you better images with it".

The idea in plain words

Anima never reads the text encoder directly. Between the Qwen3-0.6B and the DiT sits a small LLM adapter (6 transformer blocks). It takes the T5 tokens of your prompt and pulls information out of Qwen's hidden states. That adapter was trained together with the 0.6B, so it only understands the 0.6B's way of laying out numbers.

Qwen3.5-0.8B also outputs 1024-wide vectors, so it plugs in without errors, but the numbers mean something completely different. Plugged in raw, the adapter output only has a cosine similarity of 0.45 with what the 0.6B gives, and the images go off-prompt.

So I kept Anima, the adapter and the Qwen3.5 layers frozen and trained only a tiny translator on top of Qwen3.5. I call it the head: Linear, ExpRMSNorm, SiLU, Linear, about 2.1M parameters. Its only job is to turn Qwen3.5's output into something the adapter already understands.

The trick that makes this easy is where you compare the two encoders. They use different tokenizers (151k vs 248k vocab), so the same prompt splits into different tokens and their outputs don't line up position by position. The adapter output does line up: it's always one vector per T5 token, and the T5 tokens are the same for both. So I compare there:

teacher: prompt -> Qwen3-0.6B -> adapter (frozen) -> target
student: prompt -> Qwen3.5-0.8B, first 20 layers (frozen) -> head (trained) -> adapter (frozen) -> prediction
loss:    mean squared error between prediction and target

No token alignment, no DiT, no images. Just text.

What you need

  • A GPU. I used an RTX 2060 12GB and everything below fits in 12 GB.

  • The stock Anima text encoder (qwen_3_06b_base.safetensors), Qwen3.5-0.8B-Base, and an Anima DiT file. From the DiT you only need the adapter, the 118 tensors under net.llm_adapter. (I used anima-base-v1.0).

  • Code for Qwen3, Qwen3.5 and the Anima LLM adapter. I used Diffucore's. ComfyUI has all three too, and HF transformers covers the two LLMs.

  • About 25 GB of free disk for the cache.

Big warning first: make sure your Qwen3.5 code is actually correct before you train anything. My first Qwen3.5 port was wrong (and so was the reference I based it on), and everything still "worked": it loaded, ran and made images. It was just garbage underneath. Qwen3.5 is a hybrid model with Gated DeltaNet layers, and it's easy to get subtly wrong. The cheap check is to compare hidden states against HF transformers on the same weights, or measure perplexity on normal English text. HF gave me 4.7 to 9.1 on my test texts. My broken port gave around a million.

Step 1: Prompts

The head learns from prompts only, so the prompts should look like what people actually type into Anima. I made 26,200, all SFW:

  • 12,500 tag prompts from isek-ai/danbooru-tags-2024 (sfw split, posts with at least 8 general tags). Tags shuffled, 8 to 40 of them, 1 to 3 quality tags 70% of the time, an @artist tag 50% of the time, character and copyright tags, counts like 1girl up front, parentheses escaped the way Anima wants. Half of them separate the sections with a comma, half with a comma and a blank line.

  • 6,500 natural-language captions (the short LLaVA ones) from CaptionEmporium/anime-caption-danbooru-2021-sfw-5m-hq.

  • 6,500 mixed prompts: WD tagger tags from the same dataset (threshold 0.35, up to 25 tags), then the caption after a blank line.

  • 700 negative-prompt style lists (worst quality, bad hands, watermark...), because your negative prompt goes through the encoder too.

Every 25th prompt goes to validation, which gave 1,048 validation prompts.

You don't need NSFW prompts. I checked the finished head on held-out NSFW tag prompts and the error was 0.021, vs 0.017 on SFW tags. The head translates the encoder's output, it doesn't learn concepts, so it generalizes fine.

Step 2: Cache both encoders once

Both LLMs are frozen, so there's no point running them every epoch. I ran every prompt once and saved to disk:

  • Target: Qwen3-0.6B, then the adapter with the prompt's T5 ids. Shape [T5 tokens, 1024].

  • Student input: Qwen3.5-0.8B hidden states after layers 18, 20 and 22 (one forward pass, three taps). Shape [Qwen3.5 tokens, 1024].

All fp16. That's 2.79M Qwen3.5 tokens and 3.3M T5 tokens, 23 GB on disk. It ran at about 31 prompts/s on the 2060, so 14 minutes.

Why not the last layer? The last layers of a base LLM are tuned for guessing the next token, not for describing your prompt, so I tapped at 75 to 90% depth. Layers 18, 20 and 22 all ended up at the same error (0.022), so I picked 20 and cut layers 20 to 23 off completely. Smaller file, faster encoder, same quality.

Step 3: Train the head

The head in PyTorch:

class ExpRMSNorm(nn.Module):
    def __init__(self, dim, eps=1e-6):
        super().__init__()
        self.weight = nn.Parameter(torch.zeros(dim))
        self.eps = eps

    def forward(self, x):
        x32 = x.float()
        x32 = x32 * torch.rsqrt(x32.pow(2).mean(-1, keepdim=True) + self.eps)
        return (torch.exp(self.weight.float()) * x32).to(x.dtype)

head = nn.Sequential(
    nn.Linear(1024, 1024), ExpRMSNorm(1024), nn.SiLU(), nn.Linear(1024, 1024))

ExpRMSNorm is RMSNorm with the scale stored as exp(weight), so it starts at exactly 1. The layout is the same as lylogummy's Qwen3.5-4B Anima encoder, so loaders that support that one already know the shape.

One training step:

pred = adapter(head(student_hidden), t5_ids,
               target_attention_mask=t5_mask, source_attention_mask=qwen_mask)
m = t5_mask[..., None].float()
loss = ((pred.float() - target.float()) ** 2 * m).sum() / (m.sum() * 1024)

The adapter is frozen (requires_grad off) but not under no_grad, so the gradient flows through it into the head.

Settings: AdamW, lr 1e-3, 100 warmup steps then cosine to zero, batch 32, 4 epochs (3,144 steps), gradient clipping 1.0, fp16 autocast with a GradScaler. Batches are grouped by length so there's little padding. It took 14.4 minutes on the 2060.

How it went, measured on the validation prompts at the adapter output. Relative error 1.0 means "no better than outputting the average", 0 is a perfect copy of the 0.6B:

  • Raw Qwen3.5-0.8B, all 24 layers, no head: error 1.12, cosine 0.45

  • After epoch 1: error 0.035, cosine 0.980

  • After epoch 4: error 0.022, cosine 0.988

Two gotchas that cost me time:

  • With batches you have to give the adapter padding masks for both sides (Qwen tokens and T5 tokens). Before training, check that a padded, masked batch gives the same output as running each prompt alone. Mine matched to 3e-6 in fp32. If it doesn't match, you're training on padding.

  • The adapter builds big attention maps on long prompts. On 12 GB I had to turn on activation checkpointing per adapter block for the longest batches.

Step 4: Export as a normal text encoder file

I wanted a file that loads like any other text encoder. So the file is just the embeddings, layers 0 to 19, and the head saved under the name of the final norm it replaces:

out = {"embed_tokens.weight": sd["embed_tokens.weight"]}
for k, v in sd.items():
    if k.startswith("layers.") and int(k.split(".")[1]) < 20:
        out[k] = v
for k, v in head.state_dict().items():  # 0.weight, 0.bias, 1.weight, 3.weight, 3.bias
    out["norm." + k] = v.to(torch.bfloat16)
save_file(out, "qwen_3_5_08b_anima_head_L20.safetensors")

Here sd is the Qwen3.5 file with the model.language_model. prefix stripped and the vision tower and MTP weights left out. The result is 1.34 GB. Diffucore counts the layers in the file and builds the head when it sees norm.0.weight, so it just works there. ComfyUI's built-in loader expects a full Qwen3.5, which is why I wrote the comfyui-qwen35-anima-head node.

After exporting, load the file through the real loader and compare its adapter output with the training path. They should agree up to bf16 rounding. That catches a wrong key name before you waste time on images.

Step 5: Test with images, and do it blind

The numbers only tell you the head copies the 0.6B. Images tell you whether that's good enough. All my tests used cogent3_pump_rate + pump_taper, 32 steps, CFG 7, shift 3, 1024x1536.

First a quick check with 4 prompts x 2 seeds: the 0.8B with the head gave the same compositions and the same attribute binding as the 0.6B, while the raw 0.8B went off-prompt.

Then a blind test: 12 prompts full of clothing details, sides randomized, and I didn't know which image was which until the end. I picked the 0.8B 4 times, the 0.6B 6 times, and 2 were ties. Before blinding I was honestly sure the 0.8B did details better. That didn't survive. If you make one of these, test it blind, because your eyes will lie to you.

Can I do this with a bigger LLM?

Yes, the recipe works as is. Qwen3.5 has 2B (hidden size 2048, 24 layers), 4B (2560, 32 layers) and 9B (4096, 32 layers) base models. The only change is the head's first Linear, which goes from the LLM's hidden size to 1024. Tap at around 80% depth again.

But here's the catch. With this recipe the 0.6B is the teacher, and the loss literally says "copy the 0.6B". A perfect student IS the 0.6B. So the best a 9B can reach is the same quality as the 0.6B, just with more VRAM and a slower load. On top of that, the frozen adapter was built around the 0.6B, and the head can only translate into what the adapter already understands.

To actually beat the 0.6B you need a training signal that isn't the 0.6B:

  1. Train the head on images. Use Anima's own denoising loss through the frozen DiT on real images. I tried this with the 0.8B on a rented L40S: 18,046 general-rated Danbooru images (score 10+), 4,166 steps, batch 4, lr 5e-5, 1024 buckets, with a pull toward the 0.6B adapter output (weight 1) so it can't drift away. About 5 hours and $5.56. Result on 300 held-out images: loss 0.13341 vs 0.13347 for the 0.6B. Better on 61% of the images, so it's real, but tiny, invisible by eye, and it stopped improving around step 1500. With the 0.8B, the head is clearly not the bottleneck.

  2. Train the adapter too (fully or with a LoRA), with the image loss. That's the real lever, but now you're fine-tuning part of Anima: more data, more GPU, and it can end up worse than stock.

In my personal opinion, a bigger LLM only pays off with option 2. I haven't tried it. If you have the GPU, that's the experiment worth running, and I'd love to see the result.

There's also already a Qwen3.5-4B Anima encoder by lylogummy. When I tested it with corrected Qwen3.5 code, it followed my prompts worse than the stock 0.6B (lost objects, mixed up which character wears what). Its head wasn't trained to copy the 0.6B, so that's not a verdict on 4B itself, but it shows bigger isn't automatically better.

Practical notes if you go bigger:

  • Disk: the cached hidden states grow with the hidden size. For my 26k prompts, per tapped layer: 0.8B 5.7 GB, 2B 11.4 GB, 4B 14.3 GB, 9B 22.9 GB. The targets stay 6.7 GB since they're always 1024 wide. Tap one layer, not three.

  • VRAM: the cache pass only needs the layers up to your tap. In fp16 that's roughly 3.5 GB for the 2B, 7 GB for the 4B and 13 GB for the 9B. Training stays small since only the head trains. Generation is a different story: the 4B already needs full offload in Diffucore on my 12 GB card.

  • Loading: Diffucore reads the Qwen3.5 size from the file, so a 2B, 4B or 9B with a head should load the same way (I've only run the 0.8B and the 4B). My ComfyUI node is hard-coded to the 0.8B, but ComfyUI already has 2B/4B/9B configs, so it's a small change.

  • Other model families (Qwen3-4B, Gemma and so on): comparing at the adapter output doesn't care about the tokenizer, so the training works for any LLM. Your UI just has to support that architecture as an Anima text encoder.

My scripts are glued to Diffucore's model code right now. If people want them, I can clean them up and put them on GitHub, so let me know in the comments. And if you just want to try the encoder, it works out of the box in Diffucore UI v0.2.1 or newer.

0