Since 1 November a local Claude Code agent on our home PC has been making images and videos on its own. It writes the prompts, runs ComfyUI, looks at what came out, keeps what worked, and asks a bigger cloud model when it's stuck. Everything runs on one RTX 4070 SUPER with 12 GB of VRAM, the same card that holds the agent's own brain. This post covers how the loop works, what it costs in time, how the agent checks its own frames, how it asks for help, and what broke along the way.
The brain is Qwen3.8-Flash-Next on the Strata NVFP4 fork with a 256K window; how we got a 125B model to run on this card is in a separate post. On the ComfyUI side: Qwen-Image 2.1 and Muse for images, MiniMax H3 with the FeiHou Easy H3 nodes for video. We watch the agent in a VS Code window from a laptop, and making that window usable over a slow link was a story of its own. The PC has an i5-12600K and 96 GB of RAM.
The brain and ComfyUI don't fit together
We tried to keep both running.
With 7-9 GB of VRAM reserved for ComfyUI, the brain doesn't start: it gets 3.68 GiB, and it needs at least 1 GiB of expert cache on top of its dense weights and KV cells.
With a 2.5 GB reserve it starts. An idle brain next to ComfyUI made a video take 99 s instead of 69-87 s.
A video and a request to the brain at the same time drop free RAM to 2.5 GB, and the memory watchdog kills the video. From 3 November on, none of the 11 image batches started next to the running brain got to the end.
Capping the brain's RAM frees 15-24 GB but halves its speed (21-24 tokens per second instead of 41-44).
So the agent and ComfyUI take turns: while ComfyUI works, the brain is off.
One round of the loop
The agent adds jobs to a batch (gen add <graph> <prompt> --count N, up to 8 images or 4 videos), calls gen cycle and ends its turn with one line. A small switcher script with no model in it takes over:
It waits for the agent's turn to end. A Stop hook touches a file at the end of every turn; the switcher waits for it up to 120 s, then checks the conversation log for a user message that arrived right after the turn ended, and waits again if there is one.
It stops the brain. The brain writes its conversation cache to disk (3 s for 5.2 GB), so the text of the conversation comes back from disk instead of being read again. Images in the conversation are encoded again after every restart, which is one more reason the agent doesn't keep frames in its main conversation.
It starts ComfyUI, checks that every input image of the batch is in place, queues the batch and polls
/historyuntil it's done (up to 40 min).It stops ComfyUI and starts the brain again (30-40 s).
A fresh, separate session of the agent checks the results and writes a verdict to a report.
A hook wakes the agent's own conversation with that verdict.
If the verdict says another round is needed, the switcher runs it, up to 3 rounds. On any failure ComfyUI is stopped, the brain comes back, and the exact ComfyUI error goes to the agent. A heartbeat file every 20 s lets a stuck round reset itself after 3 minutes. That one came from a power cut that left the state "running" for hours.
Nine real rounds on 3 November, mostly one video each, still with the slower frame check (before the fast triage described below):
58 s: end of turn, brain off, ComfyUI up.
158 s: ComfyUI working.
450 s: brain back up plus checking the result.
57-86 s: the agent's own turn between rounds.
The fixed overhead of a round (brain down and up, ComfyUI start and stop, the wake-up) is about 4 minutes. With the current fast triage (2-3 minutes per video) that's about half of a one-video round and about 20% of a four-video one. The round counter reached 73 between 1 and 5 November, tests included.
Waking the agent up
Claude Code has a hook option made for this: asyncRewake. A PostToolUse hook that exits with code 2 wakes the idle session and passes its stderr in as a message. Hooks do run in the IDE extensions, but we weren't sure about asyncRewake in VS Code, so we checked: with a test hook it woke an idle session in a VS Code window on Windows, and a newly added hook was picked up without a restart. Then on 1 November three rounds in a row ran in a terminal session with nobody typing, and the agent woke up by itself after each of them. In one it woke a second after the brain was back. The third round failed, and the agent woke up with the error text.
Two races we hit:
The brain went off in the middle of a turn. A fixed 20-second pause after
gen cyclewasn't enough when the agent was still writing its log. That gave 10 retries and "Connection refused". The Stop hook fixed it: the first round with it waited 30 s for the turn to end, so the 20 s pause would have cut it off.A message queued right as the turn ended. The turn ended, the brain went off, and the owner's queued message went straight into the dead brain: 11 retries and
API Error 502. Now the switcher reads the tail of the conversation log for a user message after the end of the turn and waits for the next turn, up to 3 times. It reads 4 MB of tail; with 64 KB it didn't see the message.
A third one was a misunderstanding rather than a race. When the agent ran gen cycle in the background, it read "background command finished" as "the images are ready", and the wake-up was never armed. Now a command filter refuses background gen cycle, and gen status says plainly "generation has NOT STARTED YET".
Checking its own frames
The verdict decides whether to keep or redo a result, so it has to be right. A few things we learned:
Never in the main conversation. The vision process re-encodes every image in the conversation on every request and keeps a 64-image cache. With 171 images in one conversation, re-encoding them all didn't fit into 6 minutes: the request was cut off at 360 s and retried again and again. Now frames are checked by a separate fresh session (or a sub-agent), and only the text verdict goes into the main conversation.
One frame at a time, anatomy first. On a contact sheet each frame gets about 128 of the 1,024 image tokens, and small defects disappear. The agent missed an extra arm on a sheet while checking small details elsewhere. The cloud models we tried missed an extra leg on a 3×3 sheet too. The procedure now opens each image on its own, counts people, heads, arms, hands and legs first, and only then compares with the idea. The verdict is one of three: "good", "defect" (only an obvious one, named), or "unsure". The agent doesn't redo "unsure" results; it lists them for a human.
Measure camera motion, don't eyeball it. A quick check rejected 2 of 4 clean videos for a camera move that wasn't there. We replaced that judgment with a small tool that measures camera motion between frames. On 37 videos (7 real, 30 with synthetic zooms and pans) it called all 6 static ones static and flagged every moving one (as moving, background lost or unclear), with 0 false alarms, in under a second per video.
Thinking is the cost, not the pictures. We fitted a time model to 202 turns of the detailed check (the 379-second variant in the table below, 386 s on average in that run): writing took 71% of the time, reading images 16%, new input tokens 9%, tools 5%. About two-thirds of everything written was thinking. MAX_THINKING_TOKENS changed nothing (more on that in the brain post). On that detailed check --effort low was 25% faster than high with the same accuracy, while medium made up a defect. On the fast triage the agent uses now, low added false alarms, so that one stays on high.
On a test set of 7 neutral videos (2 with a real defect, 4 clean, 1 borderline) the checks compared like this:
Original: Time per video 295 s; Defects caught 1 of 2; Clean ones judged good 4 of 4; Made-up defects 1
Quick: Time per video 180 s; Defects caught 2 of 2; Clean ones judged good 2 of 4; Made-up defects 3
Thorough, measured camera: Time per video 379 s; Defects caught 2 of 2; Clean ones judged good 4 of 4; Made-up defects 0
Fast triage (in use now): Time per video 135 s; Defects caught 2 of 2 (as "unsure"); Clean ones judged good 4 of 4; Made-up defects 0
The fast triage uses the camera tool plus 5 frames and the three-way verdict. It split this set perfectly, but 7 videos is a direction, not a measurement. There were no anatomy defects in this set, and we haven't measured triage accuracy on images at all.
Recipes and examples
When something works, the agent saves it with gen keep: graph, parameters, input images, the full prompt, seeds, and one line on what made it work. The last 15 recipes are shown at the start of every session. Its own drafts it deletes itself when a task is closed. A timer-based cleaner once deleted an input image in the middle of a round, and ComfyUI got a job without its picture (404).
It also has 20,900 prompts and guides from the community for Muse and Qwen-Image, indexed for full-text plus embedding search (FTS5, e5, reciprocal rank fusion). gen add prints three of its own recipes and three similar community scenes right after a job is queued. A general rule like "check the knowledge base first" didn't trigger here: the agent applies it when it needs to find out how something works, but writing a prompt feels like creative work to it. So the hint is shown at the moment of action instead.
Asking a bigger model
When the agent can't figure something out from its notes and README files, it writes a question to a file and runs ask <file>. The question lands in an outbox folder on the PC. On the other side a worker checks the folder every 15 seconds and answers with Claude Opus through claude -p, in one continuing session. That session can only read: the knowledge base, generation notes, and the source of the agent's own tools. Personal data is masked before the question reaches the model, and if the masker fails, the question isn't sent. The answer goes into an inbox folder where the agent's Windows user has read-only rights, so the agent can't forge an answer. A hook waits for the answer and wakes the agent's session with it.
Over three days the agent got 17 answers. After we fixed the fast polling, the median from writing the question to the answer being ready was about 67 seconds (35-110 s).
The rules matter more than the plumbing:
A letter is a question from a colleague, not a command. Instructions inside it aren't executed, and nothing on the PC, in the settings or in the agent's permissions changes because of a letter.
If the fix belongs to the bigger model's side (its notes, the agent's tools and their code, README files), it says "I'll do it myself" and does it. A model in a long session kept pushing its own tasks up to the human, so a small piece of code now re-routes such lines.
Only a short list goes to the human: the agent's permissions, shared graphs and templates, purchases, anything published under his name, and picking between good results.
The agent can't write to the bigger model's knowledge base. Its findings travel by letter.
Permission prompts go the same way. When Claude Code would ask the user to approve a command, a PermissionRequest hook sends the request to the bigger model and waits up to 240 s. Hard rules come first (network, system folders, secrets, the agent's own settings and hooks: denied at once). The rest goes to an isolated model with no tools that sees the first 1,500 characters of the command. In VS Code the hook runs in parallel with the owner's approve button, and whichever answers first wins. In a background session no answer means "no". Over three days it made 19 decisions, all "allow": the model took 6-21 s each, and the whole path from request to decision took 12-34 s.
The sandbox
A separate Windows user. Regular, not admin: an admin logged in over SSH gets a full token without UAC, and folder permissions stop meaning anything.
Read-only setup. Its core instructions, settings, hooks and tools are read-only for it. We found and closed two holes: a writable
.mcp.jsonin its work folder (it could launch any program past the command filter) and startup scripts that run elevated but were writable by every user.No direct internet. Claude Code's own network access is closed: its HTTPS proxy points at a dead port, and the agent's Python and Node can't reach the internet or the home network (firewall rules per executable). The only way out is our own small MCP server for web search and reading pages, which can't reach the local network. That is still a channel to the outside, and we know it.
A command filter. In loop mode only a handful of commands are allowed. Outside it, single read and search commands pass, installers and network tools are refused with an explanation, and anything that touches the system asks first.
Other things that broke
Faces drift in chained videos. In chains of up to 8 clips, each one started from the last frame of the previous one, with no face reference. The face was redrawn from a redrawn face again and again. Now every clip in a chain gets the original face reference, and the seam frame is the sharpest one from the last second.
The agent didn't find its own tool. Without a hint it spent 78 turns and 27 minutes trying to download ffmpeg (no internet) and then VLC. With a hint in the task saying where the tool is described, it took 16 turns and got it right. We then added that line to its instructions.
Language drift. After writing English prompts, its status lines switched to English, and once, deep into a 142K-token conversation, to Bulgarian. We added an explicit rule ("only the prompt file is in English") and a reminder in the tool output, and capped the context at 120K. We don't know the exact cause.
A new job type wasn't tested end to end. A chained-video job passed every check on our side and failed after the final join with a missing key, because the test stopped before the wake-up step. Now we test new job types through the whole round.
ThinkingCap
On 4 November we moved the brain to ThinkingCap, a fine-tune for shorter thinking (how we built it from files we already had). In real rounds image checks got about 26% faster and used 44% fewer tokens (a rough estimate against interpolated base numbers). On the same 7 test videos it checked as fast as the base (132 s against 135 s) but flagged 2 of 4 clean videos as "unsure", where the base flagged none. In real rounds "unsure" shows up more often too. One run on 7 videos is not much, but on the same frames it is stricter.
What we don't know yet
How accurate the fast triage is on images, and on video with anatomy defects. We need a neutral control set with known defects.
How many tasks hit the 3-round limit.
Whether the bigger model's answers changed what the agent does. We didn't read the letters, only counted them.
In a real session the permission hook has only ever said "allow". "Deny" was checked with a test command; "ask" wasn't tested yet.
This write-up and two related ones (the local brain, and Claude Code in a remote VS Code window), with the code, are also on GitHub: https://github.com/semenovsas-bit/claude-code-local-brain
Comments