The Disappearance of Haruhi Suzumiya

Anima Base 1.0 with Suzumiya Haruhi removed by direct weight manipulation

What was done LLM write-up:

'''

Implement dataset-driven character erasure for an Anima diffusion checkpoint using direct cross-attention weight editing.

Objective:

Suppress suzumiya_haruhi while preserving Kyon, Asahina Mikuru, Nagato Yuki, Koizumi Itsuki, Asakura Ryoko, and Tsuruya. All evaluation prompts and generated images must be SFW. Do not retrain the entire model and do not modify the source checkpoint in place.

Data:

Use Danbooru 2025 metadata to build plain-text tag captions. Keep general, artist, character, and copyright tags, but remove IDs, ratings, scores, dimensions, URLs, timestamps, hashes, and other metadata. Select captions containing at least one relevant Haruhi-series character. Use thousands of real captions rather than a small synthetic prompt list.

Target construction:

For captions containing suzumiya_haruhi, create paired neutral captions by removing the target name and identity-specific correlated tags. Determine correlated tags statistically by comparing their frequency in Haruhi captions against captions for the retained cast. A reasonable criterion is:

- at least 100 target occurrences

- target/retain frequency log2 ratio >= 4

Expected identity cues include:

suzumiya_haruhi, orange_hairband, yellow_hairband, yellow_ribbon, orange_ribbon, red_armband, armband, hairband, hair_ribbon, sos_brigade, suzumiya_haruhi_(cosplay), and alternate_hair_length_(longer).

Do not remove artist/style tags merely because they contain the name; for example, preserve haruhisky.

Replace the removed identity with a neutral concept such as anime character, while retaining pose, clothing, background, composition, and unrelated character tags. Expand the target set by treating each correlated cue as the focused cue in its real caption context.

Include name aliases:

- suzumiya_haruhi

- suzumiya haruhi

- Haruhi Suzumiya

- 涼宮ハルヒ

Conditioning:

Load Anima’s actual Qwen text encoder and tokenizer rather than using CLIP or a substitute. Encode the original and neutralized captions through the same conditioning path used during Anima inference. Collect the contextual activation associated with each focused target token or cue.

Preservation set:

Build separate activation statistics for:

kyon, asahina_mikuru, nagato_yuki, koizumi_itsuki, asakura_ryoko, and tsuruya.

Balance these characters independently so high-frequency characters do not dominate the covariance matrix.

Weight editing:

Edit only the diffusion model’s cross-attention key and value projections:

diffusion_model.blocks.<0-27>.cross_attn.k_proj.weight

diffusion_model.blocks.<0-27>.cross_attn.v_proj.weight

There should be 56 edited tensors total. Leave every other tensor unchanged.

For each original K/V matrix W₀, solve a regularized UCE-style least-squares transform:

W' = (Σ αᵢ W₀ nᵢ cᵢᵀ + Σ βⱼ W₀ pⱼ pⱼᵀ + λW₀)

(Σ αᵢ cᵢ cᵢᵀ + Σ βⱼ pⱼ pⱼᵀ + λI)⁻¹

where:

- cᵢ is a target/Haruhi contextual activation

- nᵢ is the corresponding neutralized activation

- pⱼ is a retained-character activation

- α controls target-erasure strength

- β controls preservation strength

- λ stabilizes the solve

Starting hyperparameters:

- target-name weight: 7

- alias weight: 0.5

- ordinary retained-context weight: 0.25

- retained-character preservation scale: 5

- regularization: 0.5

Accumulate covariance matrices in float32 or float64, solve in float32/64, then convert each edited tensor back to its original dtype. Preserve all tensor names, shapes, checkpoint metadata, and architecture.

Verification:

1. Confirm exactly 56 K/V tensors changed.

2. Confirm all remaining tensors are bytewise or numerically unchanged.

3. Measure mapping MSE before and after editing.

4. Measure relative drift on the six retained characters.

5. Run same-seed SFW comparisons between the original and edited checkpoints.

6. Test the canonical tag plus spaced, Western-order, and Japanese aliases.

7. Test dense prompts containing Kyon and Haruhi together to ensure Kyon remains recognizable.

Treat the result as strong empirical suppression, not a mathematical guarantee. A detailed description of all visual attributes may reconstruct a similar character. Also report collateral changes—especially for characters whose original representations are entangled with Haruhi.

'''