What is it?
Geometric Memory Recall.
After a series of experiments, I have figured out how to teach distilled geometric memory recall using expert models, enforcing rules and specific guidelines for recalling memory beyond models current context limits.
A memory bank
The models themselves are frozen. Everything about the models is extracted through data accumulation, measuring, and inference mechanisms.
The core target student is utilized as an anchor, where everything must rely on the student's inputs, while the teachers are meant to add informational accumulation based on the additional complexity that their inference implicates and applies to specific boundaries of the complexity.
Simply put, it accumulates information from multiple sources, and pools them so the base model can grabble at it.
In a nutshell it's an adding machine that uses geometry to know where to look for the numbers.
geolip-bert-8192
https://huggingface.co/AbstractPhil/geolip-bert-8192
This one's context window was trained specifically with Bert's internal layers marked as anchors, and the experts feeding information into a memory basin for the system to use to recall information.
Did it work?
92.3% accuracy from ModernBert and 72% accuracy from LongFormer adjacently, both experts utilized during training in order to teach standard BertLarge's memory.
So, yeah it kind of worked. The sample size was limited, roughly 50,000 wikipedia articles, but the validation set was the measure of accuracy not the actual wikipedia articles trained.
This can be greatly expanded and improved over time, or y'know I could move to something much... larger.
geolip-clip_l-576
https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576
I trained this using ModernBert as a memory adjudication tool, so it's likely not going to work with sd15, sdxl, or flux. However, it does work.
The training result was around 94% accuracy from modernbert, so yeah the training set did imprint.
Did it work?
Yes, but with a catch. It will most likely not work in older diffusion models. However, it does work for the intended purpose, which is to test the hypothesis of context window expansion through memory recall.
So yes, it did work. It does expand the context window of both models.
How to preserve the CLIP effect
https://huggingface.co/AbstractPhil/geolip-bertenstein
I have a working prototype for what is described below, it operates on pure alignment recall through a series of INFONCE loss mechanisms with geometric mechanisms, and has an R1 recall of near 100% for multimodal structures. It will do the job, but it's a multistep process to set one up.
This one is a bit more taxing on the hardware. You need to directly align the clip-vit-l memory system with the clip-vit-l vision encoder simultaneously with an additional layer of regularization. This essentially would preserve the original text encoding system in conjunction with the vision encoding system, and this would be a pooled shared mechanism between the two.
This version does not do this, but I will likely train one tomorrow or the day after, depending how long it'll take.
What's next?
I'll probably train some sd15 loras and see how it behaves. If it works well I'll likely work out a clip_g memory pool and move to SDXL for some there.
Is there a limit?
Theoretically, this memory bank does not have a limit. It uses an entirely different set of collapse principles than other AI, while simultaneously enabling behavior that should be collapse-heavy or too difficult to reasonably learn within reasonable amounts of time.
Can this be trained Non-Geometrically to AI? Oh most definitely yes. However, this process is heavily geometric-centric, meaning the entire process revolves around an anchoring system that can't simply be superimposed into another structure without some serious testing and multiple mechanisms between where it currently is to where it would need to be.
I'm sure a competent engineer could get a linear algebraic version prepared in a minimum amount of time, but it would require a series of attention mechanism tweaks and compromises that geometry simply does not require.