I found a choice model with a perfect context window for our needs; nice and small with rope capability.

https://huggingface.co/nomic-ai/nomic-bert-2048

Only trained on a bit of stuff so it's not completely burned, which means ideal for fitting into our new structure here.

I sliced the context window into 26 77 token spaces; and am training the model specifically based on those slices and those information segments.

We are currently in stage 1; fitting and testing the fitted BERT for compliance. If it's compliant, we can move onto the next stage. If not, reassessment and potentially a larger BERT may be required.

What is BERT?

Bert is a powerful LLM built on tried and tested architectures with many innovations applied to many finetunes. This particular nomic bert 2048 finetune has a type of concept called "RoPE", which allows it to embed a type of complex positional capability within itself - thus enabling larger context windows due to a series of innovations from the team who created it.

This context window enables a 2048 space contained within it's RoPE constraints, instead of the standard BERT window.

What are we doing with Nomic BERT?

We are slicing pieces of that encoder space; and then we're giving those slices special tokens so they can enable new behaviors within the model.

Simply put, we are training Nomic BERT to be not just aware, but contextually obligated to slice up awareness into compartmentalization in the process.

77 token slices

Each of those slices are being fitted currently. Each are tasked with a certain rule;

<subject> or <texture> or whatever is deemed fit within the system. The rest of the caption space is padded and so on, enabling full saturation of the 2048 space within this model's capability.

Why though?

Our expert selector-bot will be perfect choice to target the shunt collective experiment.

The goal being; BERT will give our individual shunts better formed information based on the input prompts; selected based on BERT.

CURRENTLY BERT will only produce these segments in pieces right. Each section being meant to be devoted to something, will primarily make crap noise. However, as the training refines, I will increase the prompt lengths from maybe 15-30 tokens all the way to the 77 token threshold using carefully curated prompting selection thanks to the classifier capability that BERT has.

This is going to enable the multi-modal form, which will send that information to our shunt mechanisms through a gated temporal segmentation tool, entirely designed to select the most optimal shunts for the current generation. At first it will be simple and only select the four most likely to be the best suited candidates. This will allow the model to not bloat like a balloon at runtime, but it can also be toggled to improve fidelity and quality by enabling even more shunts at generation time.

The big shunt experiment.

It's within the grounds of being trainable as it stands, due to CLIP_L and BERT being so compliant. The internals of BERT are highly complex, enabling behavioral conditioning with this sort of detailed information and segmented conditioning.

It's a great combination experiment that could potentially yield a much more stable 2048 context window than the original shunt plan had in mind, while simultaneously enabling a full world-conceptualization system alongside it within the realm of BERT's encoder.

Set 2 of the encoders will be larger encoders, but I haven't designed the internals yet so stay tuned for that.