I haven't written any articles for a bit, so I just wanted to write something to say I'm still plugging away at ggufy and TensorPencil in my spare time.
ggufy, if you are not familiar with it, is a very simple and efficient drag-and-drop tool to convert image models to other formats, like NVFP4 or W4A8 or q6_k, in order to reduce memory requirements. It is a bit more complete so it's had less updates. There are still things I want to do with it (like text model conversion, base model conversion, creating AIO's, splitting AIO's...) but lately I've been a bit stuck on it with investigating activation-aware quantization techniques like AdaRound, SDNQ, and other similar ones.
That's been frustrating because after a lot of work and testing it seems like it is... just not very effective. The most improvement I got was about 3% (pixel error rate improvement) but that takes hours of GPU time to run through the activations to calculate that, which really just seems... not very worth it. The work there did give me a way to do on-the-fly tensor quantization, which could be helpful in calculating sensitvity files for ggufy, but even sensitivity files are not always worth it for all models, from what I can tell. Maybe I am missing something fundamental - I am, after all, still learning a lot about this stuff - but for now I've benched the activation quantization efforts in ggufy.
I have added support for H3 and SenseNova in ggufy. Minimax H3 is a video model, and SenseNova is... well, it's kind of a mixed-format model because it also has essentially an LLM embedded into it, but for our use cases it's generally an image / image edit model. It's interesting in that it doesn't use a VAE, there have been some other experiments with pixel-space generation before, but this one seems quite efficient and fast. The model itself is quite large (35GB BF16) but this is due to it kind of being an AIO model with the text encoder included, and the text encoder only needs to be loaded in the beginning, then can be unloaded to run the actual image generation. It's very good at infographic / text generation for a model of this size, it has much better capabilities here than Krea 2 from my tests, you're really able to pack a lot of text into a single image and it does pretty well with it. I've heard that it works well for comic generation due to this as well. I haven't yet experimented with it a lot, but the built-in editing capabilities should help with that as well. By default it takes around 50 steps but there is an 8-step turbo LoRA available for it. Overall I would say it's a bit rough around the edges but has potential if people can figure out how to fine-tune it effectively.
Sample SenseNova image:
On to TensorPencil! TensorPencil is a desktop program to use LLM and diffusion models together, with the ability to chat to the LLM and have it generate images, and iterate on prompts. It's still in a pre-alpha state, so it's still moving fast and changing a lot. I've done a bunch of re-architecting of the code to make it easier to add new LLM / diffusion architectures and also reduce code duplication across different backends. Warning: Technical details ahead. The native Zig compiler (Zig is the programming language ggify and TensorPencil are written in) has the ability to output directly to the language different graphics drivers speak: SPIR-V for Vulkan and PTX for NVIDIA (AMD is a goal I would like to get to eventually, but I'm not there yet). However, there are other issues with Zig's native compiler (Zig is still a very young language and is changing a lot) that prevent me from using its native compiler for the rest of the program, so I have to use the LLVM backend to do the lowering for that part. However, LLVM can't emit the SPIR-V or PTX code. So, I let LLVM do the lowering and linking, then I remove the LLVM-compiled functions for the native graphics code, compile those with the native compiler, and slap those into the executable instead to get kind of a Frankenstein's monster program from two different compilers. But it works pretty great and vastly reduces code duplication across the backends; some of those files were surpassing 10,000 lines in length.
All of that opened up some DType paths and backend paths that were previously not available, essentially for free, and ended up with LESS lines of code for the same or better result, which is a good sign that it was the right move.
Much more visible are the changes to the GUI! I hated the way it looked so I've done a bunch of work on making it more clean and usable. Still room for improvement, still iterating on the design, but it looks way better than it did previously.
TensorPencil now supports a lot more than just Krea2 for diffusion: Krea 2, SD 1.5, SDXL, Z-Image, Anima and SenseNova U1.5. The CLI also supports MiniMax H3 for videio generation, but I don't have that in the GUI yet, I have to think about how I want to fit video generation in there cleanly. It's also a lot easier now to switch between LLM models and diffusion models and it carries forward the per-family sidecar models (text encoder, VAE, etc.) which are tweakable in the settings page. Preliminary support for LoRA is also new, currently only tested on SenseNova with the turbo LoRA and H3, again for turbo, but I'm planning on expanding that to all the other models. Something that I like here is that when you select an architecture, such as Krea 2, it will only show you the text encoders, VAEs, and LoRAs that are compatible with the Krea 2 architecture, so you don't have to try to remember or guess which text encoder is used by which model.
I'm working on getting an alpha release of TensorPencil out, so other people can use it and give me feedback and bug reports. I'm hoping to have that ready this week.