EasyLLM for ComfyUI — First Test Release

An LLM that lives inside ComfyUI. It chats, writes prompts, can trigger image generation, and runs locally — no API keys, no external servers. Just point it at a GGUF file and go.

This is a first public release — it does a lot but it's still young. Some things work great, some things are rough around the edges. I'll be straight about which is which.

What it actually does

The core idea is simple: instead of alt-tabbing between ComfyUI and some chat app to write prompts, you have an LLM right there in the node graph. It's a normal chainable node — wire it up, configure it, queue it, same as anything else in ComfyUI.

There are two backends:

GGUF (EasyLLMGGUF node) — this is the main one. Uses llama.cpp directly inside ComfyUI. Fast (30-50 tok/sec with a Q4_K_M 8B model), supports vision models, has the model browser. Needs llama-cpp-python installed (there's a launcher that handles this).
CLIP (EasyLLM node) — zero-install fallback. Piggybacks on Qwen text encoders already loaded by models like Anima, Z-Image, or Flux Klein. Slower (3-12 tok/sec), no vision support, but it works immediately with no extra setup. Good if the GGUF install gives you trouble.

Interactive chat + image generation

This is the feature I'm most happy with. The LLM can decide when to generate an image — you ask for something, it outputs a structured action (generate_image, edit_image, or just_chat), and a Trigger Router node splits that into individual sockets that feed into CLIP Text Encode → KSampler.

The flow:

You type "draw me a cyberpunk cat in a neon alley"
LLM outputs JSON with a prompt and negative prompt
Trigger Router breaks it out to separate sockets
Those wire directly to CLIP Text Encode → KSampler
An Image Capture node saves the result with session info
The frontend auto-queues the pipeline
Image shows up in your chat history

You can follow up with "make it more colorful" and it adjusts and regenerates. Works with most models in generate mode.

There are working example workflows included:

InteractiveChat + Generate.json — chat with single image generation
InteractiveChat + Generate + Edit.json — adds multi-turn refinement with an edit group
Enhancer.json — single-shot prompt enhancement wired to KSampler

These workflows were tested with Flux Klein 4B but the pattern should work with other models — you'll just need to swap the model/CLIP/VAE nodes.

Prompt enhancement

Set the node to Enhancer mode and it transforms short ideas into detailed prompts, wired straight to CLIP Text Encode. There are 18 built-in templates organized by category — enhance (Standard, Creative, Creative+, Expert, JSON, Universal), transform (Expand, Compress, To Natural Language), describe images, edit images, and interactive modes like Chat+Generate.

Input: "a mystical forest"
Output: something like "A mystical forest at twilight, ancient oaks with glowing runes, ethereal fireflies, moss-covered stone pathway, volumetric fog, cinematic lighting"

Nothing revolutionary, but the convenience is real — no context switching, no copy-paste, the output flows directly into your pipeline.

Vision (image understanding)

The GGUF backend supports vision models — LLaVA, Qwen-VL, Gemma-3-Vision. Drop a mmproj file next to your model and it auto-detects. You can upload images directly from the chat popup (no Load Image node needed) or wire one to the image socket.

Useful for: "describe this image", "what art style is this", "suggest a LoRA for this look". The description output can feed into CLIP Text Encode for img2img-style workflows.

Vision-capable models get a badge in the model browser. The mmproj matching is automatic but not perfect — you may need to name files consistently.

Prompt library

A system prompt manager with 18 built-in templates (all editable), plus you can create your own, drag to reorder, search by name. Import/export as JSON files — this is where the Civitai sharing angle comes in. Export your best enhancer presets or character templates as .json and share them alongside workflows.

Model browser

Basic but functional. Shows your GGUF files, lets you switch models quickly, marks vision-capable ones. Not the most polished thing — it's a simple file browser — but it saves you from typing paths manually.

Image capture with metadata

The Image Capture node saves generated images with ComfyUI workflow metadata embedded, plus it catches the positive prompt text and embeds it in a way CivitAI can read. If you upload images to CivitAI, the prompt info should carry through.

Text utility nodes

Seven lightweight nodes for cleaning up LLM output before it hits CLIP Text Encode: Text Input, Joiner, Replacer, Extractor, Limiter, Whitespace Cleaner, Duplicate Remover. Zero dependencies, chainable. Useful when the LLM output needs some cleanup.

Chat database (early days)

There's a history database that stores conversations and generated images. Exports to Markdown. It works, but it's basic right now — I have ideas for making it more useful but for now it's mostly a persistent chat log.

Installation

There are launcher scripts for Windows (.bat), Linux (.sh), and macOS (.command). They auto-detect your GPU (CUDA, Vulkan, Metal, CPU fallback) and install the right pre-built wheel. No C++ compiler needed. If something goes wrong there's a recovery menu with fallback options.

If you're using ComfyUI Manager, install.py is auto-discovered during node install.

If the GGUF install fails for whatever reason, the CLIP backend works immediately with Qwen-based models — so you can still use most features while you sort out the llama.cpp situation.

Important if you use Ollama: Ollama stores models in a proprietary format. You need the original .gguf file from HuggingFace. Qwen2.5-7B-Instruct-GGUF with the q4_k_m quant is a good starting point — fits in about 4.8GB VRAM. If you already have GGUF files for LM Studio, those work directly.

Honest assessment

What's solid:

GGUF inference at 30-50 tok/sec
The chat → generate → capture pipeline
Vision model support (with compatible models)
Prompt library with import/export
Text utility nodes
Launcher scripts and GPU auto-detection

What's work-in-progress:

Some GGUF model architectures may have quirks — chat template detection isn't perfect for every model
The model browser is basic, does the job but nothing fancy
The database/chat history is functional but under-developed
Vision support depends on having a compatible mmproj file; not all vision models behave the same
The chat UI works but will evolve based on feedback

What this isn't:

Not a replacement for ChatGPT/Claude — it runs small local models
Not a polished product — it's a v0.x test release
Not guaranteed to work perfectly with every GGUF file out there

If you try it and something breaks, I want to hear about it. Bug reports and workflow shares are what will shape where this goes next.