KM-MIX Unexpected product

我所有的研究都依赖直觉，而得到的这些东西又太过抽象，甚至难以理解和表述。我不知道研究是不是从出发点就有问题，也没有多少头绪，只能不断的试错。与其说是研究，倒不如说是穷举，和暴力破解解密游戏的密码一样没品。

距离上一次更新过了很长时间，所以还是同步一下近期的研究结果。

虽然理论方面有很大的突破，但在实践中还是失败了，依旧没有很好的混合多个LORA。

好了，现在我希望你能对模型和LORA进行一些抽象的理解，不然我很难去解释这些研究结果。

LORA的本质是一段记忆，是模型训练LORA时对图片的认知。

只有很少数LORA能达到知识这个阶段。

模型确实和人的思维在某些地方有着共同处

人类 观察 记忆 想象

模型 训练 标注 生成

好吧很扯不是吗，我都没有力气继续了。

All my research relies on intuition, and the results are so abstract that they’re hard to understand or articulate. I don’t know if the starting point of the research itself is flawed, and I don’t have many clues, so I just keep trial-and-erroring. Calling it research feels generous—it’s more like brute-forcing, as tasteless as cracking a password in a decryption game.

It’s been a long time since the last update, so let’s sync up on the recent research results.

While there’s been a big breakthrough in theoretical research, the practical side has still failed. I haven’t managed to effectively mix multiple LORA.

Now, I’d like you to develop an abstract understanding of models and LORA, because otherwise, it’s tough for me to explain these research results.

At its core, a LORA is a piece of memory—a model’s cognition of images when it’s trained on a LORA.

Only a tiny fraction of LORA reach the stage of being "knowledge."

There are indeed some similarities between models and human thought.

Humans: observe, remember, imagine.

Models: train, annotate, generate.

Yeah, it sounds pretty out there, doesn’t it? I’m already too exhausted to keep going.

模型层级以及LORA的信息层级

我想你们在使用图生图测试LORA时，也发现了提升LORA权重时的图形变化规律。这一点也可以通过对一个模型不断的重复拆解流程进行复现，基于此发现我将模型划分为 A~E 5个层级。

初版的拆解模型大致属于C层级基础模型 和 破损A层级画风模型，这个组合不能对模型进行完全活化，并影响了模型对LORA的解读和获取

但层级E很特殊，如果A~D层级是模型理解的信息，那E层级就是模型的潜意识，是模型训练中未被标注的信息。好吧，这一切都太抽象了，而LORA也有类似的表现。

附加解析LORA时，如果LORA的信息层级大于模型活化层级时会造成后续混合的数值溢出。小于模型活化层级时LORA的权重可以设置的很高，比如9999.

模型混合获取时，如果LORA的信息层级大于模型活化层级时，模型会突出显示LORA信息。小于模型活化层级时则会正常获取LORA信息。

Model Hierarchy and LoRA Information Hierarchy

I believe you’ve noticed the pattern of graphical changes when increasing LoRA weights during image-to-image testing of LoRAs. This can also be reproduced by repeatedly dissecting a model’s processes. Based on this, I’ve divided models into five hierarchies: A to E.

The initial dissected model roughly falls into a combination of a C-tier base model and a damaged A-tier art style model. This combination fails to fully activate the model and affects its interpretation and acquisition of LoRAs.

However, the E-tier is quite special. If tiers A to D represent the information the model consciously understands, then the E-tier is the model’s subconscious—information that wasn’t labeled during training. Admittedly, this is all very abstract, and LoRAs exhibit similar behaviors.

When additionally parsing LoRAs, if the LoRA’s information hierarchy exceeds the model’s activation hierarchy, it causes numerical overflow in subsequent blending. If it’s below the model’s activation hierarchy, the LoRA’s weight can be set extremely high, like 9999.

During model blending, if the LoRA’s information hierarchy is higher than the model’s activation hierarchy, the model will prominently highlight the LoRA’s information. If it’s lower, the model will acquire the LoRA information normally.

暂时先同步这些吧，其余的研究并没有得到确切证实，只是一些想法和猜测，以及个人的感觉。毕竟研究也快2年了，不能和早期那样发布不成熟的理论和模型了。

后续研究方向。

1.通过削减模型层级，从而制作新一代拆解模型。

目前已经能做到完美获取LORA信息，但要想获取更多LORA信息的话当前的拆解模型还是有所不足。

尝试制作了几个削减模型但问题一个比一个离谱。

如果能完成提取模型，那研究就终于能够结束了吧。

2.不断的进行模型混合堆叠LORA信息，观察结果，或许量变能够产生质变。当然也有可能到达SD1.5的极限

3.混合E层提取模型，观察是否有信息聚合现象。这是一个假设的现象，因为我使用的所有素材都来自2年前，所以模型的表现也应该属于同时代的表现，但事实并不完全如此。

4.固化模型，模型混合的信息多次重复会导致模型认知固化，设想中这可以固定模型的主题。

For now, let’s sync on these points. The rest of the research hasn’t been conclusively verified—just some ideas, guesses, and personal feelings. After nearly two years of research, I can’t keep releasing immature theories and models like in the early days.

Future Research Directions:

1.Create a new generation of dissected models by reducing model hierarchies.

We can now perfectly acquire LoRA information, but the current dissected models are still insufficient for extracting even more LoRA data.

I’ve tried creating a few reduced models, but each one has more absurd issues than the last.

If we can complete the extraction model, maybe this research can finally come to an end.

2.Continuously blend models and stack LoRA information to observe results.

Perhaps a quantitative change could lead to a qualitative leap. Of course, it’s also possible we’ll hit the limits of SD1.5.

3.Blend E-tier extracted models to observe potential information aggregation phenomena.

This is a hypothetical phenomenon. Since all the materials I’ve used come from two years ago, the model’s performance should theoretically align with that era, but that’s not entirely the case.

4.Stabilize models.

Repeated blending of model information can lead to cognitive solidification in the model. In theory, this could lock in the model’s thematic focus.

总之，研究好麻烦，混合实验，测试分类，什么的都好麻烦，好累，每天坐在电脑前10多个小时，还没有电脑椅。没时间外出，没时间玩游戏，整天给图片找不同，模型混合次数都8万了，眼睛度数也越来越大。说是搞科研，谁信啊，自己都不信。我看到和AI相关的信息都难受，觉得自己做的所有事情都毫无意义，只是在浪费时间。虽然什么一个人搞科研的故事听起来很酷，很有趣，和文学作品里的变态科学家一样，但我只想说：“哈~哈~NO!”。总之不要再有下一次了！

以下是一些研究成果的演示页面

Here are some demonstration pages of research findings

模型拆解研究页面 Model Deconstruction Research Page

https://civitai.com/models/579793

LORA提取演示页面 Lora extraction demonstration page

https://civitai.com/models/1086230