Today is the day I begin it at full force.
I slammed head first into an impasse point., where I must decide if I want to adopt an additional 4 days work on comfyui to fully integrate all the models as reusable visual encoders using the gross internal structure - requiring every single integration a bitch-and-a-half up-step to get into my architecture, which defeats the entire core purpose of this structure to begin with - or if I want to create the mentioned proper loader, handler, and off-loader nth diffusers structure for one week, and then save hundreds - or potentially thousands of hours for teams who decide to adopt it as a quick solution for prototyping before they go into full bottom level integration.
I decided, it's time to put on the programmer socks and get to work. They're made of steel chainmail.
Today I establish the new implicitly controlled and server-secured paradigm for handling nth vram, ram, and offload capable model hooks within comfyui and the architecture. Everything will have a direct conversion point node, and everything converted will be converted directly to a form that the baseline ComfyUI can use it's various methods of insanity to communicate with other models.
This task of building specifically to snap into current nodes with a simple conversion - is no easy task at all. It will be done however, and it will exist in the shunt suite until I format it into a reusable and modular form for use.
I'm starting a new branch for this one called vram-dev, which will merge into dev as soon as the vram testing system is capable - preserving dev in it's current state for testing and debugging at frozen 0.8.0
https://github.com/AbstractEyes/comfy-clip-shunts/tree/dev
I may have gone a little overboard with dev as-is, and it should be merged into main before I continue, but it's unstable so I need to codify the replacement loader, vram handler, and offloader with the care required of an expert surgeon blacksmith. I can't leave this to chance, not if I want it to properly work with huge models on potato pcs.
I ended up just calling it dev2_electric_boogaloo in a typical in the moment spur. TYPE THE WHOLE THING!
Once it's prepare; it will be portable to everything and a codified abstract library for reuse, until then it will exist on dev-vram.
This version is meant to have VRAM aware capability that can house any torch tensor information from any broadcast point to any point within the program, codified and separated based on simple calls and checks - left open ended enough for the developer to rationalize through a simple enough interface to make sense in torch terms, along with sense through comfyui node.
This is a uniformization of comfyui's vram structure ported directly to a diffusers archetype and paradigm. I've been working on this for quite some time in my more cognoscente hours, and today it presents itself as a requirement - not just a "will happen" or "soon to be".