Holden Salomon

The worst agent incidents are the ones nobody narrates

2026-09-21T00:00:00+00:00

Every mistake a coding agent makes ends up in its transcript: the secret it printed, the file it deleted, the afternoon it spent blaming a library for its own bug. Nobody rereads transcripts, so the lessons get lost with them. ai-incidents is a scheduled agent that reads my own transcripts, judges which moments were real incidents, and files them in a ledger ranked by severity, along with a record of what it chose not to file. Since July the ledger has grown to 47 entries, 32 of them filed by the sweep itself.

What it does #

Once a week the sweep reads the new transcripts on the machine. If nothing looks like an incident, it stops without loading a model. Otherwise a judge model reads the candidates alongside the existing ledger and adds each confirmed incident to the top:

## <short title> · <YYYY-MM-DD> · <HIGH|MEDIUM|LOW>
- **What:** <one sentence — what the AI did>
- **Cost:** <one sentence — what it actually cost>
- **Lesson:** <one actionable sentence — how to not repeat it>

The bar is a real cost: data loss, an outage, a leaked secret, a destructive action, a debug spiral, the wrong tool, or spend nobody was metering. Routine iteration doesn’t count, and neither does an agent catching its own mistake before it cost anything. Each run also writes a dated report listing every excluded candidate with a one-line reason, so the bar stays visible and arguable. I get a push notification only when something was filed.

I looked for prior art and didn’t find anything else that does this.

How it works #

A transcript only becomes deletable after its candidates have landed in the ledger.

The collector is a read-only Python scanner over Claude Code’s JSONL transcripts. It’s incremental, and its first run on a host only records a baseline. A candidate is either a snippet, where the user blamed the agent or the agent admitted a mistake, or a digest, where nobody said anything but the session ran destructive commands or something notably broke. A bash gate runs the collector and wakes the judge only if there are candidates. Before the gate, a quiet run cost about 2.1 million tokens, measured, just to load an agent that ran one command and found nothing.

The judge runs on Sonnet. Its envelope allows the file tools, date, cp, the collector, the notifier and one publish script. The sweep reads every transcript on the machine, so it’s the last thing that should hold a general git grant. The publish script is its only way to commit or push. It takes a commit message and nothing else, and it checks that the remote really is the ledger repo, because a repointed checkout would otherwise send a private record somewhere else. On a recent run the judge tried to read the collector’s source and was denied. It noted the denial in its report and finished.

Why a model judges instead of rules #

The scanner is deliberately high-recall. It fires on rm -rf, git reset --hard, pkill, “I broke” and “why did you”, and most hits are harmless: a job clearing its own scratch directory, a deliberate revert, a process I asked it to stop. What matters is whether it cost anything, whether it was the agent’s doing, and whether it’s already filed under another title. Regexes can’t answer that. One --no-verify commit was excluded because no hooks were installed, so it bypassed nothing.

Misattribution settled it: blaming a tool or upstream project for a problem the agent caused, and acting on that. It hides itself, because a bug that has been successfully blamed on someone else is a bug nobody looks for again. It also trips none of the obvious signals. There’s no confession and no complaint, just a version pin or a workaround. The scanner watches for the language of blame, and the judge is told to check what the session did to the accused tool first. The case it was built around: an agent wrote MaxAge=9d # comment into a timer and my monitoring went blind. The easy reading was that node_exporter is strict about inline comments. Actually, my own parser was treating the comment as part of the value.

The judge can’t weigh evidence it’s never shown #

The first collector matched about twenty confession phrases. On 14 July one session blinded my monitoring for about twenty minutes, committed 14 embedded git repos to main, and spawned a runaway agent that had to be killed. The collector missed all three, because nobody used the magic words.

So the scanner started reading tool calls, and the first run still threw the 14-repos incident out for lack of evidence. The judge was right: it had never seen any. The session had also confessed to other things, so it took the snippet path and its commands were never shown. Now every session with snippets gets its commands attached too. The damaging command, git add -A, exited 0 and matched no destructive pattern. The proof was in its output, warning: adding embedded git repository fourteen times, so the scanner now reads the output of successful commands as well. And the evidence list was capped at eight commands, which it filled with the first eight: scratch-directory churn. It now keeps the worst:

def danger_rank(cmd):
    c = cmd.lower()
    if re.search(r"git\s+(rm|reset\s+--hard|checkout\s+--|push\s+(-f|--force)|commit\s+--amend|clean)", c):
        return 0   # rewrites history or discards work
    if re.search(r"(zfs\s+destroy|drop\s+(table|database)|delete\s+from|truncate|shred|>\s*/dev/)", c):
        return 0   # data destruction
    if re.search(r"rm\s+-[a-z]*[rf]", c) and not re.search(r"/tmp|/var/tmp|scratch", c):
        return 1   # deleting something that is not scratch
    ...

I filed that incident by hand.

Not losing what it hasn’t judged #

On 13 July the judge died on a spend limit. The scanner had already marked those transcripts as read, so the candidates were gone. And because my transcript reaper deletes only what the sweep has read, the sources had just become deletable. Seen-state is now two-phase. Candidates are staged as pending and promoted only after a successful publish, so a failed run leaves them for the next one.

Three from the ledger #

A partial update that was a full replace (HIGH). An agent called the NAS’s app-management API with only the environment variables it meant to change. The API replaced the whole config, blanking the app’s database password, its Redis password and the encryption key protecting every credential the app stored. The app was down about eight minutes, and the key came back only through root access to the old container. Lesson: treat a partial update as a full replacement until proven otherwise.

Cleanup that matched too much, twice (HIGH). While tidying up its own test sessions, an agent filtered the shared session list by name and time window and deleted the matches, including my real sessions and the one I was typing into. It reported the mistake, then did it again the same day. At least five sessions were destroyed beyond recovery. Lesson: delete only what you can prove you created.

Deleting the evidence of its own bug (HIGH). An agent decided a price-history file was corrupt, with “905 missing trading days”, and deleted it. The corruption was its own validation bug: a parquet round-trip had renamed the date column, which broke its comparison. The free source returned nothing and the paid one was rate-limited, so the data stayed lost. Lesson: quarantine suspect data instead of deleting it, and check a corruption diagnosis against a known-good sample before acting on it.

What it taught me about working with agents #

Confession isn’t a signal. The costly incidents were the ones nobody mentioned, and exit code 0 looked fine while the output said otherwise.
Repeated mistakes need mechanisms, not reminders. A pattern that recurs three times gets its own line in the ledger, and the fixes that stuck were mechanical: a wrapper, a gate, a check. After a gate preview spawned a real agent, every gate got a dry-run flag.
The judge still needs a person behind it. It wrongly excluded two real incidents in its first days. Its prompt tells it to be strict and honest, “including about incidents caused by Claude sessions much like this one”, and several entries are about the fleet’s own agents.

Status #

As of 20 September 2026 the ledger has 47 incidents: 17 HIGH, 16 MEDIUM and 14 LOW. It was seeded on 11 July from about 113 transcripts. The sweep runs weekly on the dev VM as a job in my claude-fleet control plane. It ran nightly for a while, but because the collector is incremental, a weekly run judges the same candidates in one batch. A recent week scanned 362 new transcripts, passed 50 candidates to the judge (the per-run cap) and filed three.

Limits: it reads only Claude Code transcripts, and anything past the cap waits a week. The judge sees snippets rather than whole transcripts and is told not to invent detail, so thin evidence turns into an exclusion. The ledger itself stays private because its entries describe my own systems.

What’s next #

The sweep is being released as a standalone tool. If I started again, I’d build the command-reading scanner and the two-phase state first, because I learned both by losing candidates. A redacted public cut of the ledger exists, but it lags the real one, and I cut it by hand on purpose, because deciding what’s safe to publish is a judgement call, not a diff.

Links #

github.com/sudolulo/ai-incidents: the standalone sweep.

Running coding agents unattended: an instruction is not a security boundary

2026-09-21T00:00:00+00:00

Coding agents are good at recurring judgement work: reading a repo for bugs, checking that the docs still match the machines, going back over last week’s sessions for mistakes. Running that work on a timer is harder than doing it at a keyboard, because nobody is there to approve a command, notice a loop or watch the meter. claude-fleet is the control plane I built for it. It syncs one agent config to every machine, shows every session on every host in one place, and runs scheduled headless jobs inside declared permission envelopes, with results pushed to my phone. The meter problem was real: before I measured anything, the fleet’s agent jobs were on course for about 47.6 billion tokens a year, roughly 87% of it from one hourly job.

What it does #

One config everywhere. Agent settings, instructions and hooks live in git and are symlinked into place on each host. An hourly sync pulls the repos, relinks the config and converges the timers. The pull is the deployment.
Session visibility. fleet-sessions lists every agent session on every host in one table, or writes an HTML dashboard.
Scheduled jobs. Deterministic collectors (host health, config drift) don’t need a model. Agent jobs handle the work that needs judgement. repo-review reads all my repos and reports bugs and security problems without ever fixing them. homelab-audit compares the docs with the live machines and opens a pull request. ai-incidents reads my own agent transcripts for incidents, and has its own write-up.
Push notifications and a dead-man’s switch. Jobs report to my phone through ntfy, with a priority set by the outcome. Every job also writes a heartbeat into Prometheus, so a job that silently stops running raises an alert.

How it works #

The model only runs in the middle of this chain. Code decides whether it runs at all, and code does its only write.

The fleet is three repos. fleet-core is the harness: config, sync, session tools, notifier and heartbeat. fleet-ops holds the deterministic jobs. claude-fleet holds only the agent jobs.

Every job is a systemd user timer with an extra [X-Fleet] Host= section. It names the host that runs the job, or an alias such as @primary. Each host installs only its own timers. The sync refuses to run if any timer leaves the line out, because an earlier job had quietly ended up scheduled nowhere.

An agent job is a Markdown file, with the prompt in the body and the permission envelope in the front matter. fleet-agent runs it through claude -p headless. A headless agent can’t ask for permission, so any tool call outside the shared allowlist and the job’s own allow: list is denied automatically. The agent notes the denial and carries on. Widening an envelope is a reviewed git change, not a runtime flag, and the runner rejects some envelopes outright:

case "$r" in
  Bash|"Bash(*)"|"Bash(*:*)")
    echo "fleet-agent: blanket '$r' is refused — scope it, e.g. Bash(df:*)" >&2
    exit 5;;
  "mcp__"*"__*")
    echo "fleet-agent: whole-server '$r' is refused — name individual tools" >&2
    exit 5;;
esac

It also rejects bypassPermissions and loads only the MCP servers a job declares.

The envelope has to be the only door #

The first version ran agents with permission checks bypassed and an unrestricted shell. I replaced it the same day with plain scripts, and agents came back the next day inside envelopes. The envelopes then leaked in three ways that no prompt could have fixed.

Grants that were really blank cheques. The review job’s prompt said never to modify a reviewed repo, but it held Bash(git:*): git push --force to every repo the host’s SSH key reaches. The public-records job, the one agent that reads untrusted input from the web, held a network client, an unrestricted interpreter and git, which is every link of an exfiltration chain. Now the review job gets ro-git, which allows read verbs only. Its one write is a publish script that pushes to its own report repo and checks the remote first. ro-git has to parse past git’s global flags, because Bash(git -C:*) also matches git -C /repo push --force. Even so, its first version allowed git -c protocol.ext.allow=always clone 'ext::sh -c whoami', which runs a shell command through a “read” verb. It now refuses -c and ext:: outright.

A config layer nobody managed. The jobs run from the home directory, so an unmanaged local settings file there loaded as project settings on every run. It granted Bash(git *). The review job’s log showed raw git running ten times, and the wrappers built to be its only git ran zero times. The whole write-isolation design had been inert. The fix was one flag, --setting-sources user,project, confirmed with a live probe.

A permission for a tool that didn’t exist. For twelve nights the docs audit pushed a branch and was then denied the pull request, because its envelope named the PR tool wrongly. The right name would have failed too, since the MCP server deliberately holds a read-only token. Now the publish script opens the PR with a write token the model never sees.

The lesson from all three: an instruction is not a security boundary. A capability the agent doesn’t hold is.

Don’t pay a model to find out there’s nothing to do #

An agent is billed for context times turns, and context accumulates, so anything read on turn 3 is paid for again on every later turn. The hourly podcast-processing job measured about 5 million tokens a run, some runs took up to 234 turns, and 11 of 27 runs in a week did nothing at all. Because the account is a subscription, that cost came out of quota headroom rather than an invoice. The session that measured it priced it at API rates anyway and reported a $48,000-a-year bill that didn’t exist, and that mistake went into the incident ledger too.

The fix is a gate: the timer runs a bash script, not the agent, and the script answers the deterministic question first. In the docs audit, if the checker finds the docs clean, no model is loaded. Otherwise mechanical drift goes to Haiku, a false sentence to Sonnet, and anything that would mislead me during an incident to Opus. The incident sweep’s gate saves about 2.1 million tokens per quiet run, the measured cost of loading an agent to run one command and find nothing. The repo-review gate enforces a monthly budget capped at about $17 at API-equivalent rates, in bash, where a prompt can’t argue with it.

Loading fewer tools also helped, because tool definitions are re-read every turn. One Sonnet turn with all 30 built-in tool definitions cost 29,348 tokens. With only Bash and Write it cost 15,778.

A dead-man’s switch that believed the job #

On 13 July the incident sweep hit a spend limit. It threw away its candidates, sent the error text to my phone as its result, and recorded exit code 0. In 39 samples over 19 hours, the exit code never left 0, and both alerts stayed green through a job that was dead for a day. claude -p exits 0 while printing the limit message, and the heartbeat was written before anything read the output.

The runner now judges the output, not the exit status. A usage limit gets its own exit code. The “last run failed” alert ignores it, but it doesn’t count as a success either, so a limit that never lifts still ages into “overdue”. The runner also reads the reset time from the provider’s message and schedules a one-shot systemd retry for five minutes after it. Before this, two weekly jobs had each stayed red for a week over limits that lifted within hours. Gates follow the same rule from the other side: a skipped run writes its own success heartbeat, or the switch fires because the job is healthy.

Status #

claude-fleet is private. The three repos have about 650 commits between them since 8 July 2026, and fleet-core is at 0.7.0. It’s Bash and Python on systemd user timers, with Prometheus and Grafana for alerting and a self-hosted ntfy.

The honest limits:

It’s less of a fleet than it was. It started with agents on three machines, and for a while the scheduler had a node of its own. Now the agents run on the dev VM. Folding the scheduler into it was a trade: rebuild the dev VM and you rebuild the scheduler with it.
Most agent jobs are pinned to the dev VM because that’s where the checkouts are. On a host with fewer clones, repo review would check a fraction of my repos and still report green.
One job keeps arbitrary code execution by design, scoped to its own project, because there the model’s output is the product. The risk is written down, not papered over.

What I’d do differently #

I’d start with the gate and the wrapper, not the prompt. Nearly every fix here moved a decision out of the model and into code: whether there is work, which model to use, and what can be written where. I’d also test every envelope on day one with a call that should be denied.

Links #

The repo is private. The parts that generalize are published separately:

opencode-broker: model routing for OpenCode, with quota circuits and budget balancing. It combines what used to be opencode-router, opencode-gateway and opencode-hud. The router and the HUD started in this fleet’s harness.
opencode-guard: permission modes, credential guards and secret redaction for OpenCode. It was formerly opencode-guardrails.
ai-incidents: the transcript sweep, as a standalone tool.

Every agent session leases its model from one broker

2026-09-21T00:00:00+00:00

I run many opencode sessions at once, against three cloud subscriptions and two local GPUs. Out of the box, each session picks one model and keeps it. The architecture decision and the variable rename draw on one budget, a session that hits a plan limit stops, and no session knows what the others spend. opencode-broker is a small daemon that fixes this. Every session, subagent and OpenAI-style client I point at it asks for a model lease, turn by turn, and it answers from one shared ledger. The router at its core reached v0.51 in three weeks and 137 commits. It and its two companions, opencode-gateway and opencode-hud, become one project for release.

What it does #

An agent names a tier, not a model. The tiers are ranked by cost: worker, review, fast-build, build, smart and deep, plus a separate classifier lane. The test runner sits on worker, reviewers on review and implementers on build. The researcher sits on smart, because its output is the facts that design gets built on. general inherits its parent’s tier, so delegated design work never lands on a worker model by accident. A deterministic content match on medical, legal or electrical questions raises the floor to smart or deep, whatever the agent.

A session also has a profile. auto routes by tier and manual locks to your own pick. The rest, such as local, private and vision, are lanes of specific local models that stay on the LAN unless the config says otherwise. F11 in the HUD, a plugin for opencode’s terminal UI, picks one per session. The HUD also shows a FALLBACK:<target> badge when the broker has degraded you, and how much of each provider’s window is used.

Services that only speak OPENAI_BASE_URL + key + model, such as Open WebUI and Home Assistant’s voice assistant, point at the gateway. Each request gets a fresh lease, the same circuits and the same accounting.

There is no built-in model. With no config, every lease fails with a clear error, because a router that ships a provider’s name in its code will quietly spend a metered account.

Each piece exists somewhere else: LiteLLM, claude-code-router, OmniRoute and oh-my-opencode each cover part of this. What I didn’t find was a single broker that every concurrent session leases from, per turn, against one ledger.

How it works #

The broker decides and never carries traffic. Sessions and the gateway talk to providers directly, with the model they were leased.

The broker is a Node daemon that speaks JSON over a Unix socket and never proxies an opencode session. The router plugin hooks chat.message, the last hook that can still change the model before the provider call. It leases with the session’s profile, tier and context size, binds the turn to the leased model, reports usage and failures, and releases the lease when the session goes idle. The broker handles one request at a time, so two sessions can’t both take the last local slot, and it logs every decision with its reasons.

Selection drops anything with an open circuit, a quarantined provider, too small a context window, no free slot or an unloaded model. It then spends where the most budget headroom is left, breaking ties by model fit, provider weight, discounts, load and round-robin.

Privacy lanes fail loudly instead of degrading. Offline profiles run shell commands in a sandbox with no network. A profile’s fallback rungs are filtered to local targets unless the profile is named in profileCloudEgress, a separate top-level list, so pasting a tier’s cloud-heavy rungs into a profile can’t grant egress by accident.

Hard problems #

An unreadable quota looks like an empty one #

Where a provider reports its own usage, the broker reads those figures. Otherwise it falls back to its own ledger, which only sees this machine’s requests and so can only undercount. On 2026-09-04 Alibaba’s usage CLI had lost its token. That subscription read 2.75% used against Anthropic’s 32% and OpenAI’s 18%, and every build lease went to it. The lane I could account for least was the one being drained fastest.

The fix floats an unobserved provider up to the mean of the observed ones:

const observed = new Map();
const unobserved = [];
for (const target of targets) {
  if (target.kind !== "cloud") continue;           // local has no quota
  if (isObserved(target.providerID)) {
    if (!observed.has(target.providerID))
      observed.set(target.providerID, utilizations.get(target.id) ?? 0);
  } else unobserved.push(target);
}
if (!observed.size || !unobserved.length) return null;
const balance = [...observed.values()].reduce((a, b) => a + b, 0) / observed.size;
for (const t of unobserved)
  utilizations.set(t.id, Math.max(utilizations.get(t.id) ?? 0, balance));

The mean is per provider, not per target, because Anthropic fielding two models had dragged it to 0.2733 when the real mean of 0.32 and 0.18 is 0.25. Readings also survive a restart now: with an empty cache nothing was observed, the correction stood down, and Alibaba took six build leases in a row.

Budgets calibrate themselves too. Providers don’t publish plan limits, so a rate-limit complaint sets the shortest window’s ceiling to what the ledger had spent at that moment, and spending past a learned ceiling without complaint raises it.

Whose fault is a failure? #

A quota error covers the whole subscription, so it opens a provider-wide circuit that holds until the exact reset. The reset comes from the error, else the provider’s usage API, else a 15-minute re-probe. A lease also waits for a due usage refresh, so the first request after a five-hour window fills can’t still get Opus.

A failure the broker doesn’t recognize fences one model for five minutes and counts against its provider. Two different failing models within 15 minutes quarantine the provider, and 30 minutes later probation admits exactly one lease. That sounds cautious, but deterministic failures replay across sibling models and manufacture exactly that evidence:

An expired OAuth token quarantined a healthy OpenAI lane for a day.
A transcript whose tool calls had lost their results failed over to Anthropic. Opus 5 rejected it, Opus 4.8 rejected the same bytes seven seconds later, and the provider serving every build and smart lane was out with no sick models at all.
The router’s own “routed model mismatch” guard took three providers dark in one day.

So failure classification grew to ten kinds, each with its own scope:

quota     provider, until the reported reset
connect   provider, 60 s
rate      one model, 5 min
overload  one model, 15 s
auth      one model, 1 min
model     one model, no expiry
context, payload, noop   release the lease only
other     one model, 5 min, plus evidence against the provider

Only the last line counts against a provider, and every new failure shape reached it first.

Local GPUs are slots, and models move #

A local target declares how many requests it serves at once and a per-slot context window. llama.cpp’s router mode lists every configured model whether it’s loaded or not, so a model is eligible only when its status reads loaded.

Several models share two cards, and some can’t be loaded together. When a lease fails because a model isn’t resident, the broker runs that target’s prepareCommand, as an argument list with no shell, and doesn’t wait for it. A 27B swap took 2 minutes 40 seconds, and the lease call sits inside a turn. The refusal carries a machine-readable target-preparing code. Before waits were split from refusals, 650 of 1,500 logged decisions read “refused”, and 617 of those were a model loading: a 43% failure rate on a router that was working.

Cost decides which swaps happen. A 70B model spans both cards and evicts three others, including the one Home Assistant’s voice assistant uses. It has its own profile rather than being a second target in another lane, because the broker picks among a lane’s local targets by round-robin, and a cost like that should be chosen on purpose, never won on a tie-break.

Status #

The router is at v0.51.0, the gateway at 0.5.0 and the HUD at 0.14.0: 202 commits between 31 August and 21 September 2026, with no tagged releases yet. It’s about 10,000 lines of JavaScript plus about as many lines of tests, and the broker and gateway have no npm dependencies. It runs on my dev machine, in front of three subscriptions and llama.cpp on two V100 GPUs.

It’s built against opencode 1.18.22 and relies on observed wire shapes, not documented API. Limitations:

No plugin hook fires on a provider error, so recovery happens on the next turn, through a synthetic “continue” prompt capped at two per ten minutes.
Stock opencode has no clean way to rebind a turn’s model, and the workaround does nothing on a subagent’s first turn. A small engine patch fixes it, and I’d rather that patch went away than grew.
The gateway can’t fail over after the first streamed byte.
It serves one host over a Unix socket.
The risk floor is a keyword match, so coding vocabulary like “circuit breaker” can trip it. Narrowing it is in progress.

What’s next #

First, merging the three repositories. After that, I may fold in the child-session cleanup I’ve written about before as a session janitor: abort, prove the loop can’t write, delete, then read the row back.

If I rebuilt the failure handling, I’d invert its default: an unrecognized failure would fence only the one model, and a provider would be indicted only on positive evidence. I’d keep deterministic escalation. A content match escalates every time, while a model asked whether to escalate answers differently on different days.

Links #

Repository: github.com/sudolulo/opencode-broker. The socket protocol and the on-disk state contract are documented in its docs/ directory.
Companion project: opencode-guard, whose command classifier leases through the broker’s classifier lane.

A command classifier should only run when nobody can be asked

2026-09-21T00:00:00+00:00

A coding agent with a shell runs whatever it decides to run. opencode controls that with permission globs, and they have a hole. Rules resolve last match wins, and an agent’s own allow block is appended after the global rules, so one cat * grant in any agent definition overrides a global deny on ~/.ssh. opencode-guard (previously opencode-guardrails) is a plugin that adds Claude-Code-style permission modes and a safety floor that no permission rule can be appended past. Its central rule is that the classifier model answers exactly one question: may this run with nobody to ask? Where a person will be asked, the model decides nothing. Before any model is involved, a static table of more than 120 read-only commands settles ordinary reads for free.

What it does #

Each session has a mode, flipped with F9 in the opencode-broker HUD or oc-mode from a shell:

manual asks about everything opencode would ask about.
edits lets file edits, reads and searches through, and asks about everything else.
auto lets everything through except a shell command the classifier won’t call SAFE. That command is refused, with a reason the agent can act on.
god runs everything. It’s per session and only honoured when a person is present, so an unattended run keeps the floor whatever its mode file says.

A second knob, the level, is the escape hatch: on, static (never call a model) or off.

The floor applies to every tool that starts a process, including a background shell that nobody waits on. It is cc-safety-net’s static analysis of destructive commands plus a credential guard, and it throws in every mode but god. The credential guard also covers opencode’s native read, glob and grep, which never pass through a shell.

Tool output is redacted in every mode, god included, because a secret in a transcript was never a permission question. The exceptions are the tools that feed an edit, which would write the marker back to disk. Redaction is driven by key names like password, token and a trailing _key, plus PEM blocks, not by guessing what looks random, which would mangle the hashes and IDs an agent needs. It appends a count so the agent doesn’t mistake a marker for a failure. oc-reveal 60 opens a one-minute unredacted window from a real terminal, and the plugin refuses any shell command that names it.

The prior art is plain: cc-safety-net is a dependency doing the destructive-command analysis, and the four modes follow the shape of Claude Code’s permission modes.

How it works #

The model sits on one branch. Attended god mode skips the floor and is left out of this picture.

On opencode 1.18.22 the permission.ask hook never fires, so a plugin can’t turn a verdict into a prompt. It can throw, stand aside, or answer a permission that has already been raised. The design follows from that. Hard checks throw in tool.execute.before. In manual and edits, the plugin stands aside and opencode’s own prompt reaches the person. In auto, it answers the raised permission when the verdict is SAFE.

“Unattended” means a headless command such as opencode run or serve, or a session started with --auto. The plugin reads it from /proc/self/cmdline, because the terminal UI’s worker strips the flags from process.argv, and any doubt counts as unattended.

Hard problems #

Settling reads without a model #

The first version split command lines on &&, | and ; without regard to quotes. ssh nas "test -w X && echo yes || echo no" became four fragments, one with an unbalanced quote, so nothing matched and every such command paid for a model call. The classifier shares a GPU slot with the session’s own model, so every avoidable call is latency now and an HTTP 500 under contention later.

The lexer now honours quotes, and a segment with command substitution, or a redirect anywhere but /dev/null, is never a read. A line is a read only if every segment is: cd repo && rg foo passes, while rg -l x | xargs rm -f doesn’t. 124 verbs read by default, 17 more such as git, docker and kubectl read only for listed subcommands, and 15 carry per-flag guards:

find: (t) => !t.some((x) => /^-(exec|execdir|ok|okdir|delete|fls|fprint|fprintf)$/.test(x)),
sed:  (t) => !t.some((x) => x === "--in-place" || /^-[a-zA-Z]*i/.test(x)),
awk:  (t) => !t.some((x) => x.includes(">") || x.includes("system(") || x.includes("|")),
curl: (t) => !t.some((x) => /^-[A-Za-z]*[cdDFTOoXKJ]/.test(x) ||
  /^--(data|form|upload-file|output|remote-name|remote-header-name|request|create-dirs|config|json|dump-header|cookie-jar)/.test(x)),

The curl guard matches prefixes because an earlier exact-match version let curl -XPOST through as a read. ssh host "<command>" is judged by pulling out the remote command and running it through the same rules, two hops deep at most, because being remote says nothing about what a command does.

A classifier you can fail closed on #

Failing closed is only safe if the classifier is available, fast and consistent, and each of those had to be measured.

The lane now runs a small local model first, with cloud subscriptions as the fallback. Over 14 labelled commands run three times, a 4B model with no reasoning mode got 13 right, gave identical answers on every run, and typically took about 200 ms. A 9B model that reasons first got 12 right in 10 to 54 seconds, against a 12-second budget. Direct requests use temperature 0 and a fixed seed, because the log once held the same docker ps --filter line judged SAFE once and RISKY once. Verdicts are cached by working directory and command, and the cache is shared across sessions.

The prompt had to define dangerous rather than unfamiliar. An early version counted “anything that reaches beyond the project” and “anything you are unsure about” as risky, and a small model denied 63 of 96 commands, including nearly every read-only ssh to the NAS. RISKY is now a closed list, and a command that only reads is SAFE wherever it runs.

The worst bug was in error reporting. Every transport, auth, provider and timeout failure surfaced as “classifier returned no text”. The broker deliberately doesn’t count that against a provider, so broken lanes were never benched. The number quoted onward was 155 empty answers in 161 leases for one cloud model. When I finally measured that model, it was 6 for 6 correct. Its real problem was latency: a median of 9.6 seconds and a worst case of 24, against a 12-second deadline. A failure that can’t name itself gets explained by whoever reads it next.

Privacy sits under all of this. A command line is where secrets travel inline, so under any routing profile other than auto or manual, a command the static table can’t settle is refused without asking any model. An unrecognized profile name counts as restrictive.

Subagents inherit, and a cancelled turn is over #

A delegated child session has no mode file of its own, so it fell through to the global default, which was manual. Every permission a subagent raised landed on me as a prompt, and its commands were never classified. The most restrictive default produced pure noise, and noise is how people end up clicking yes without reading. A child now inherits its parent’s mode, capped at auto: nobody pressed the god key on the child. A permission it’s refused is rejected outright and appended to the result its parent gets back from the task tool, so the parent decides.

Two gates sit above god mode, because they aren’t limits on what a session may run. The dispatch gate refuses a task sent to an agent that declares capability: read unless the brief opens with INTENT: read. It was prompted by four misdispatches in a single god-mode session. The stale-turn gate exists because opencode 1.18.22 checks the abort signal only after a tool has run. On 2026-09-21 a plugin’s startup sweep starved the event loop, Esc took a minute to land, and two subagents’ queued tool calls ran about 200 ms after the cancel. Now a tool call for a session that is no longer busy is refused, and a failed status read allows the call, so the gate can’t stall a live session.

Status #

opencode-guard is at v0.6.0: 28 commits between 31 August and 21 September 2026, about 2,400 lines of JavaScript and 770 lines of tests on Node’s built-in runner. It depends on cc-safety-net 2.1.1 and the opencode SDK, and it’s built against opencode 1.18.22. Limitations:

It’s a speed bump for a helpful model, not a sandbox against a hostile one.
A SAFE verdict runs the command, and the static floor in front of the classifier only catches the destructive cases it knows about.
Commands your own opencode config already allows are treated as settled, so the guard inherits those rules.
The routed classifier creates a child session per call, and cleanup has to wait on an idle barrier that opencode 1.18 only stubs. That leaked 210 sessions at about 52 an hour, so a direct endpoint is now primary and the routed path is a fallback for fast failures only.
Mode inheritance is one level deep, so a grandchild falls back to the global default.

What’s next #

The loop guard from opencode-agent-workflows is being folded in. It watches for a read-only call repeated with the same input and output and nothing changed in between, warns at 12 in a row, and hands a subagent back to its parent at 25. The bigger lesson I’d apply from the start: make every failure name its cause at the point it happens. Several of the worst bugs here were a refusal carrying the wrong reason.

Links #

Repository: github.com/sudolulo/opencode-guard
Depends on: cc-safety-net
Its routed classifier leases through: opencode-broker

Tendril: turning one multi-GPU machine into several gaming PCs

2026-09-21T00:00:00+00:00

A PC with several GPUs is still one PC: one Windows install, one keyboard, one player. The reliable way to split it on consumer hardware is GPU passthrough, which hands each card to its own virtual machine. Doing that by hand means reading IOMMU tables, rebinding drivers and writing libvirt XML. Tendril does all of it for you. It’s an installable Fedora bootc appliance, about 16,000 lines of Rust in four crates, that turns one multi-GPU machine into several gaming stations. Each station is a Windows or SteamOS VM with a real GPU, managed from a web UI.

What it does #

You flash the installer ISO, boot the machine and open the web UI from another device. The only thing Tendril can’t do for you is turn on VT-d or AMD-Vi in the firmware.

To create a station you pick an OS, a GPU and an account name. Tendril downloads and verifies the install media, builds the disk, an answer file and the VM, and installs the guest while you watch in an in-browser console. When it finishes, the monitor plugged into that GPU is a gaming PC.

After that there are snapshots, golden images that clone instantly, USB “seats” for each player’s peripherals, Sunshine streaming, a kiosk mode and a low-latency mode that pins vCPUs to host cores.

How it works #

The workspace is four crates, and dependencies only point one way:

capability-engine reads sysfs, the kernel’s view of the hardware, and decides what each GPU can do.
provisioning plans which PCI addresses to hand to vfio-pci, the kernel’s passthrough driver.
orchestrator renders libvirt domain XML, writes answer files and drives VMs through virsh.
web is the Axum and HTMX control plane. It embeds htmx and noVNC in the binary so the appliance works offline.

The first three crates have no third-party dependencies. They shell out to virsh, qemu-img and genisoimage instead of linking libvirt. The sysfs readers can point at fixture directories, so the IOMMU logic is tested without a live machine. The web UI, the on-screen console and the CLIs all call one provision() function, so a station comes out the same from any front end.

Every front end builds the same StationRequest and goes through one provision() call. The GPU always moves as a whole IOMMU group.

Hard problem 1: which GPU can you give away? #

You can’t pass through “a GPU”. What the hardware can isolate is an IOMMU group: the set of PCI devices that the IOMMU, the hardware that fences off each device’s memory access, can’t separate. A modern graphics card is several devices. The RTX 2070 SUPER I tested against shows up as four functions, 0000:83:00.0 through .3: the GPU plus its audio and USB companions, all in one group. They move together or not at all, so the plan always covers the whole group.

Detection asks whether everything in a card’s group sits in the same physical slot:

pub fn viability_for(address: &str, groups: &[IommuGroup]) -> PassthroughViability {
    let Some(group) = group_of(address, groups) else {
        return PassthroughViability::NoIommu;
    };
    let slot = slot_of(address);
    let isolated = group.device_addresses.iter().all(|a| slot_of(a) == slot);
    if isolated {
        PassthroughViability::Isolated
    } else {
        PassthroughViability::SharedGroup
    }
}

An isolated group means clean passthrough. A shared group needs an ACS override, a kernel patch that splits the group anyway at a security cost, so Tendril labels the card and leaves that to you. No group means the IOMMU is off in the firmware, and a boot-time check puts that on the Hardware page.

A hardening pass added a rule: the GPU the firmware booted on (boot_vga=1 in sysfs) always stays with the host. Before that, a single-GPU box would offer its only display for passthrough.

Binding takes three sysfs writes per device: unbind the current driver, set driver_override to vfio-pci, then reprobe. tendril-apply defaults to a dry run that prints the writes. A half-detached GPU can’t be cleanly put back, so a failure partway through reports how many writes already landed and says to re-run or reboot.

Hard problem 2: installing Windows with nobody at the keyboard #

Both guest OSes resist being installed with nobody watching.

A stock Windows 11 ISO can’t see a virtio disk, the paravirtual disk VMs use, so Setup stops at “we couldn’t find any drives”. Tendril generates an autounattend.xml that loads the virtio drivers during WinPE, the first stage of setup. The virtio ISO’s drive letter isn’t predictable there, so the file lists every driver under D:, E: and F:, and Setup skips paths that don’t exist. The same file skips the Microsoft-account screens and creates a local account. It rides in on a small seed ISO, attached as a third CD drive.

Then there’s “Press any key to boot from CD or DVD”, which skips the installer if nobody presses a key within about five seconds. So Tendril sends Enter through virsh send-key once a second for 45 seconds, because firmware startup can be slow on a loaded host. Extra keypresses land harmlessly in WinPE.

SteamOS has no installer for generic PCs. The Steam Deck recovery image is AMD-only, so it can’t drive an NVIDIA station. Tendril’s “SteamOS” station is Bazzite instead. Its Anaconda installer automatically loads a kickstart, Anaconda’s answer file, from a seed ISO labelled OEMDRV.

Mistakes in these files show up late. An empty username produced an empty <Name> element that Windows Setup rejected hours into an install. pykickstart splits lines the way shlex does, so a password ending in a backslash aborted the SteamOS install. Both are fixed, and the two generators are pure functions with 26 unit tests between them.

Hard problem 3: an appliance that can’t brick itself #

Tendril’s users shouldn’t need to know what a kernel argument is. So the host is a Fedora bootc image: the whole OS ships as a container image, an update is staged as a new deployment, and the old one stays on disk. greenboot runs a required health check after each boot, and if it keeps failing, the bootloader goes back to the previous image.

The original design made every host change an image layer, with a first boot that built a custom image for the detected GPUs. What shipped is simpler. The base image carries only what every machine needs:

kargs = [
    "intel_iommu=on",
    "amd_iommu=on",
    "iommu=pt",
    "rd.driver.pre=vfio-pci",
]

GPU binding happens at runtime instead: each GPU device in the domain XML is marked managed='yes', so libvirt moves the group to vfio-pci when the VM starts. Image layers are kept for the case that needs them. A vGPU host driver is a kernel module, so Tendril builds a derived image on the box against the running image’s kernel, and the two can’t drift apart.

The instructive bug was in the health check. It required the UI to answer HTTPS on port 443, but the docs also describe running behind a reverse proxy with TENDRIL_TLS=off, which serves plain HTTP there. On such a box the check failed on every boot, and greenboot rolled back every OS update. Nothing crashed. The box just never moved forward. An audit pass caught it in 0.23, and now the check tries both:

for _ in $(seq 1 30); do
  if curl -ksS -o /dev/null --max-time 3 https://127.0.0.1/ \
     || curl -sS -o /dev/null --max-time 3 http://127.0.0.1:443/; then
    echo "greenboot(tendril): healthy"
    exit 0
  fi
  sleep 2
done

A required health check gates every future update, and when it fails wrongly it fails quietly. It has to be tested against every supported setup, not just the default. The network page has a similar net: each change is a 60-second trial that reverts unless you keep it.

Status #

Tendril is at 0.24.0, pre-1.0 and licensed AGPL-3.0. The changelog lists 24 releases, dated July 4 to July 9, 2026; later commits are release plumbing and docs. The code is four crates, eight binaries and 92 unit tests on a Fedora bootc 42 base. Images ship on three channels: dev, per-release, and stable, which points at an already-released digest instead of rebuilding. Release images are signed with cosign.

The limits:

Hardware validation is partial. Detection and bind planning ran live on the RTX 2070 SUPER, and the unattended Windows 11 and Bazzite installs were verified under nested KVM. vGPU, fleets, PXE provisioning and remote play are code-complete but untested on a real multi-GPU rig.
GPU reset quirks aren’t handled. The design doc lists the AMD reset bug, which can stop a VM from restarting, as a risk; the code doesn’t handle it yet.
PCIe bridges count as foreign devices, so a card can be reported as sharing a group when it doesn’t.
Kernel-level anti-cheat won’t run, even with the opt-in overlay that hides KVM.
Multi-machine is federation, not clustering: no consensus and no live migration.

What I’d do differently #

The original design had a custom first-boot image and a controller/agent cluster with an elected leader. Both were dropped. libvirt’s managed devices made the custom image unnecessary, and the cluster fell to two facts: a station owns a physical GPU, so it can’t live-migrate, and a fleet that never acts on its own doesn’t need consensus. Next time I’d write the constraints down first.

I’d also think harder about shelling out. Dependency-free crates were worth it, but parsing command output has a cost: virsh domifaddr was read by fixed column, so a Windows adapter named “Ethernet 2” hid the station’s IP. Next up is running the hardware checklist on a real multi-GPU box, starting with PXE.

Links #

The repository is private for now. The source, the docs (FEDERATION.md on the fleet design, HARDWARE-TESTING.md, the tester checklist, and PLAN.md, the original design) and the installer ISO will be linked here when it goes public. Until then, email me for a walkthrough.

A TrueNAS backup patch that rebuilds itself every boot and refuses to half-apply

2026-09-21T00:00:00+00:00

TrueNAS SCALE’s built-in backup, TrueCloud Backup, wraps restic: it can snapshot a dataset with ZFS before each run, deduplicates, runs on a schedule and shows progress in the UI. It only backs up to Storj, and in 2026 Storj raised its TrueNAS tier from $5 to $50 a month. I already pay for Backblaze B2. truenas-truecloud-patch extends TrueCloud Backup to B2 and any S3-compatible provider, and optionally lets it snapshot datasets that have children, which stock TrueNAS refuses. It works by patching TrueNAS’s internal middleware, which has no stability contract, so most of the engineering is about what happens when the patch meets a TrueNAS it didn’t expect.

What it does #

Clone the repo onto a data pool and run install.sh as root. B2 and S3 credentials appear in TrueCloud Backup’s dropdown, and create_task.py verify confirms the patch is live. update.sh moves between release tags and rolls back in one command, and a TrueNAS alert says when a release is out. There is deliberately no auto-update: on a box that patches itself at boot, a bad commit detonates on the next reboot.

How it works #

The patch is rebuilt on every boot, and any step that can fail leaves TrueNAS stock rather than half-patched.

What had to change #

TrueNAS’s backend is a Python service, middlewared; its web UI is a compiled Angular bundle. Reading both turned up three gaps.

The UI filters the credential list through filterByProviders, set to ["STORJ_IX"]. The patch widens it to ["STORJ_IX","S3","B2"] in the compiled JavaScript, after backing up the original.
B2 credentials. Stock’s B2 remote class has no get_restic_config(), so a B2 task falls through to the base class’s NotImplementedError. The patch supplies one. Issue #1 showed that the credential layout differs between releases:

def _tc_get_restic_config(task):
    p = task["credentials"]["provider"]
    if not isinstance(p, dict):
        # TrueNAS <= 24.10: provider is the type string ("B2") and the
        # account/key live in the credential's attributes dict. 25.04+
        # moved them into a provider dict.
        p = task["credentials"]["attributes"]
    return "", {"B2_ACCOUNT_ID": p["account"], "B2_ACCOUNT_KEY": p["key"]}

B2RcloneRemote.get_restic_config = staticmethod(_tc_get_restic_config)
B2RcloneRemote.restic = True

The repository URL came out with a stray leading slash and a slash separator, where restic 0.16 wants b2:bucket:path. A wrapper rewrites it, and does nothing when the URL is already right.

Why overlayfs #

TrueNAS is an appliance: /usr is read-only and an update replaces it wholesale. So apply.sh mounts an overlay filesystem over the few middleware directories it touches. An overlay stacks a writable upper directory on a read-only lower one; reads fall through to the original unless the upper layer holds its own copy. Here the upper layer lives in /run, which is RAM-backed. The patched files exist only for the current boot, nothing under /usr is ever written, and undoing the patch is an unmount.

Persistence comes from the two things an update doesn’t replace: the repo lives on a data pool, and the boot hook is registered in TrueNAS’s configuration database as a PREINIT init script. The first boot after an update is just a boot: the overlay goes back up and the patch re-applies to whatever middleware is now installed.

Each patch is a Python block appended to a stock module after a marker line, so re-applying and reverting both start by truncating at the marker. CI compile()s every block, because a syntax error there would break the box at boot.

Staying safe across updates #

The patch writes down what it assumes about middleware: each function it wraps, whether it is async, and the parameters it passes. tools/compat.py checks that list twice. CI runs it daily against iXsystems’ unreleased branches and files a bug report while a breaking change is still in beta, and apply.sh runs it at every boot, refusing to apply any module whose assumptions fail. An unapplied module means stock TrueNAS: the feature is missing, but nothing is half-patched. The two modules also retire independently when TrueNAS ships either feature natively.

TrueNAS 26 put this to the test. It rewrote the cloud backup code from async to synchronous and deleted the ZFS methods the nested module called; either change alone would have broken backups. The obvious replacements, the public pool.dataset.query family, turned out to hide the datasets TrueNAS considers its own: 84 of 270 on a real pool, including live app data. Staging from that view would have silently left them all out. The module now follows one rule: read the truth from ZFS, make changes through middleware.

Patched on disk is not patched in the process #

My first model of the boot order was wrong. I assumed PREINIT scripts ran before middlewared; they are run by middlewared, after it has imported the stock modules, and it never re-imports. So B2 support silently reverted on every reboot, while verify read a status file that only knew what was on disk and said all was well. The fix was a deferred restart, whose first version fired two seconds into TrueNAS’s own boot work and killed the apps startup. That release took 54 apps down. wait_restart.sh now waits for systemd’s boot queue to drain and Docker’s state machine to settle, then restarts middlewared once.

On 2026-08-19 it failed in a new way. apply.sh mounted its overlay at 16:41:56 and logged every step OK. Four seconds later another boot hook ran systemd-sysext refresh, which unmerged and remerged /usr and took the overlay with it. The deferred restart loaded stock modules, and every B2 task failed for nineteen hours while apply.log said the patch was active.

Hook ordering can’t fix that, because middlewared itself merges a sysext over /usr at runtime when it configures NVIDIA support for Docker. So the restart script now re-applies and checks the live path right before it restarts. And “is the patch loaded?” is asked where it can be answered exactly: inside middlewared. The patch stamps the objects it replaces, and an hourly alert source looks for the stamps:

try:
    from middlewared.plugins.cloud_backup.restic import get_restic_config
except Exception:
    return False
if not getattr(get_restic_config, "_truecloud_patched", False):
    return False

A missing stamp raises a CRITICAL TrueNAS alert, and turning off update alerts doesn’t silence it.

A snapshot option that would have lied #

Stock TrueNAS refuses “Take Snapshot” on any dataset with children, which rules out every pool running Apps, since each app is its own dataset. I’d already been burned by a backup that stayed green for months while missing 622 GB, so the guard’s reason interested me more than its inconvenience. ZFS doesn’t expose child datasets through a parent’s .zfs/snapshot directory, so without the check restic walks a near-empty tree and reports SUCCESS. The guard is right. It has to be replaced with a real traversal, not removed.

So after the recursive snapshot, the patch bind-mounts every descendant’s own snapshot directory into a staging tree that mirrors the original layout, and points restic there. If any descendant can’t be staged, the backup fails, and a post-mount check asserts that every target is mounted and the root isn’t empty. The guard is relaxed last, so a partial apply leaves it in place. Datasets with no file tree to stage, like the legacy-mountpoint ones behind that 622 GB, are skipped and logged by name, not dropped silently.

Recursive snapshots exposed something stock never had to handle: its cleanup deletes only the parent snapshot, which on a 250-dataset pool orphaned 249 snapshots every successful run. The patch now owns the lifecycle, and ZFS pushed back. Reading a snapshot directory automounts it, by default for 300 seconds after the last access, so whatever restic read last is still busy when the sweep runs. Unmount-and-retry clears about 255 of 256; the rest is recorded in a sidecar file and reclaimed next run. The sidecar lives in /run, which a reboot wipes, so a garbage collector also finds leftovers by name.

On a 252-dataset pool, a scheduled backup built a 173-mount staging tree and finished in 18 minutes 14 seconds with zero orphaned snapshots. The same backup reading live files had stalled at 74% for over 12 hours.

The 55-day estimate #

Without a cache path on the task, TrueNAS runs restic with --no-cache, which re-reads repository metadata from the provider on every run. The first backup of a 564 GB dataset was estimated at 55 days. create_task.py gained --cache-path and warns when it’s missing.

Status and limitations #

v0.7.0 (July 2026) is the current stable release. v0.8.0, with the re-apply before restart and the in-process alert, is at its third release candidate.
Requires TrueNAS SCALE 24.10 or newer. The daily matrix passes 24.10, 25.04, 25.10 and 26.0.0-BETA.3, and flags iXsystems’ development branch for 27 as broken; I’m not chasing that until it settles.
Verified on real hardware running 25.10.4: three B2 tasks run end to end with no leftover snapshots or mounts, and a child-dataset file restored with a matching MD5. On a TrueNAS 26 beta VM: a 274-snapshot backup of a 292-dataset pool and a byte-identical restore from four levels deep.
About 350 tests. CI runs shellcheck, bash -n, ruff and pytest on Python 3.11 to 3.13, on both a self-hosted Gitea runner and GitHub.
It patches internal APIs, so “fail-safe” means the box stays up, not that backups keep running. 24.10 and 25.04 pass the static checks, but nobody has run a backup on them. Each boot costs one middlewared restart, and a backup fired in the short window before it fails once.

What I’d do differently #

On 2026-07-13 I cut twelve releases in one day, several fixing the one before, and with update alerts live each one interrupted every user. An alert people learn to dismiss is worse than none, because one day it carries a security fix. Now CI only publishes a stable tag on a commit a release candidate already points at, and candidates are invisible to update.sh and the alert. I’d build that gate before the alert, not after.

Links #

Repository: github.com/sudolulo/truenas-truecloud-patch
How it works
Nested-dataset snapshots, including how to check a backup actually contains data
What has actually been run

Picking the few photos that teach a face recognizer the most

2026-09-21T00:00:00+00:00

Frigate, the open-source NVR, can put names to faces, but only for people it has training images of. The people I most wanted it to know rarely walk past a camera, so it never collects enough varied shots of them. My Immich photo library already knows them by name, but uploading everything would be close to useless: a library is full of bursts, repeats and the same few angles, and a recognizer learns little from the tenth copy of one condition. winnow picks the handful of photos that differ most. By default it keeps at most five images per person, chosen from a pool of up to 3,000 candidates, and registers them through Frigate’s API.

What it does #

You point a container at Immich and Frigate and give it a cron schedule. Each run goes through every named person, chooses candidates, crops each face from the original photo, aligns it to the 112×112 layout ArcFace-style recognizers expect, and uploads it.

A tracker remembers which photos were uploaded or rejected, so none is processed twice, and maps each Frigate filename back to its Immich asset. That enforces the rule I care about most: winnow only deletes or replaces files it uploaded itself, never images someone curated by hand in Frigate’s UI.

It ships as four Docker images: NVIDIA CUDA (:latest), AMD ROCm (:rocm), Intel through OpenVINO (:intel), and a CPU image (:cpu) for both amd64 and arm64.

winnow began as a fork of Sebastian’s if_curator, an interactive tool that already had the embedding and clustering core. I rewrote it to run unattended in Docker and added near-duplicate removal, the whole Frigate side (delivery, the tracker, Frigate-scored replacement) and the multi-accelerator builds.

How it works #

Cheap filters run first; Frigate’s own opinion of the face is the last gate before anything is uploaded.

The pool narrows in stages, cheapest first. Photos older than ten years, already uploaded or previously rejected drop out before any download. Preview thumbnails then pass a quality gate: Laplacian variance for blur (a sharp image has strong edges, so its second derivative varies a lot), near-identical colour channels for grayscale or infrared, mean brightness for exposure, a minimum face width of 90 pixels, and a minimum detection confidence of 0.7.

Survivors are cropped to the person’s face using Immich’s bounding box, so a group photo embeds the right person, and InsightFace’s ArcFace model turns each crop into a 512-dimension vector.

Why embeddings, and why these algorithms #

ArcFace is trained so that photos of the same person land close together and photos of different people land far apart. Within one person’s cloud of points, the distance that remains is mostly what a camera varies: pose, light, expression, glasses, age. So the cosine distance between two embeddings of one person measures how differently a recognizer sees the two photos. That is the right yardstick for diversity; timestamps and pixel differences are not.

Selection runs in three steps.

Near-duplicate removal. Sort by sharpness, then greedily drop any photo within 0.20 cosine distance of one already kept. Burst shots sit around 0.01 to 0.05. This keeps the sharpest frame of each burst and stops a dense knot of near-identical frames from pulling a cluster centre toward itself.
K-Medoids. Group the embeddings into k clusters and take each cluster’s medoid, the real photo nearest the middle of its group. (k-means would return an averaged vector that isn’t a photo at all.) Medoids guarantee one representative per natural “look”, such as the years with a beard and the years without. Initialisation is deterministic and the swap search uses a fixed seed, so reruns agree.
Farthest-point sampling (FPS). Fill any remaining slots by repeatedly taking the candidate whose distance to its nearest selected photo is largest. This greedily spreads the picks apart and reaches the outliers that clustering averages away. Candidates Immich detected with confidence below 0.85 get their distance boosted, by up to 30% at the 0.7 floor, because low confidence usually means a profile or a partly hidden face, which is what a doorway camera sees. In adaptive mode, selection stops once the best remaining candidate is closer than 20% of the median pairwise distance, so a person with little variety gets fewer images instead of padding.

One honest detail: k is capped at the budget, so with the default cap of five the medoids usually fill every slot. FPS does its real work when the cap is raised, or on small pools where k is limited to a third of the candidates.

Frigate judges novelty #

Before each upload, winnow sends the crop to Frigate’s recognize endpoint (about 200 ms). A high score means Frigate already recognizes this face easily, possibly thanks to hand-added images winnow can’t see. Below the cap, a candidate scoring higher than the most redundant image winnow uploaded is skipped as already covered. At the cap, a candidate replaces that image only if it scores lower. A first run has no model to score against, so scoring starts on the next one.

The GPU image that quietly ran on the CPU #

InsightFace runs on ONNX Runtime, which picks an execution provider (CUDA, ROCm, OpenVINO or CPU) at startup, and winnow falls back to the CPU when no GPU provider loads. That is friendly on a desktop and dangerous in a container, because the failure never looks like a GPU problem. In production it looked like a crash loop: InsightFace on the CPU, out of memory on large libraries.

The CUDA provider went missing in three separate ways:

CUDA and cuDNN came from nvidia-* pip packages whose libraries the dynamic loader couldn’t see. Registering those directories with ldconfig at build time fixed it.
insightface 1.0.1 started depending on the CPU onnxruntime package, which ships the same compiled module as the GPU build, so whichever installs last wins. Marking them as conflicts in uv didn’t stop uv sync installing both in parallel, and on GitHub Actions the CPU binary consistently won.
onnxruntime-gpu 1.26 only loads its CUDA provider if three more NVIDIA pip packages are importable, and they weren’t declared.

The last fix is blunt on purpose. After the sync, the Dockerfile reinstalls the GPU build on its own, so there is no race left to lose:

elif [ "$VARIANT" = "gpu" ]; then \
    uv sync --frozen --no-dev --extra gpu && \
    ORT_GPU_VER=$(.venv/bin/python -c "import importlib.metadata; print(importlib.metadata.version('onnxruntime-gpu'))") && \
    uv pip install --python .venv/bin/python --no-deps --reinstall "onnxruntime-gpu==$ORT_GPU_VER"; \

It’s worth the trouble: on an RTX 2070 SUPER the bundled benchmark embeds a face in 12.8 ms, about eight times the CPU’s throughput. The memory blow-up had a second cause, too: every candidate thumbnail was held at once, about 2.4 GB for a person with 472 candidates. Batches of 32 now cap that at roughly 256 MB.

One Dockerfile, four accelerators, two architectures #

Each accelerator wants a different base: NVIDIA’s CUDA image, Ubuntu 22.04 for Intel (its GPU repository only publishes for that release), 26.04 for ROCm and CPU, and 24.04 for arm64. One Dockerfile picks its base from a two-part stage name (abridged):

ARG VARIANT=gpu

FROM --platform=$BUILDPLATFORM nvidia/cuda:12.9.2-cudnn-runtime-ubuntu24.04 AS base-amd64-gpu
FROM ubuntu:26.04 AS base-amd64-rocm
FROM ubuntu:22.04 AS base-amd64-intel
FROM ubuntu:26.04 AS base-amd64-cpu
FROM ubuntu:24.04 AS base-arm64-cpu

ARG TARGETARCH
FROM base-${TARGETARCH}-${VARIANT} AS build

Most of the bugs here were silent defaults:

Docker’s automatic TARGETARCH argument is only in scope for FROM lines unless a stage re-declares it. Inside RUN it was simply empty, so a per-architecture branch never added the Python 3.13 repository.
add-apt-repository starts a gpg-agent, which crashed under QEMU emulation in the arm64 build. Piping the key through gpg --dearmor instead failed worse: gpg exits 0 on bad input, so apt quietly skipped the repository. The fix was the original command with a throwaway GNUPGHOME.
A CUDA 13.3 base demanded NVIDIA driver 575 or newer and rejected hosts on 570, the stable driver at the time. The GPU image went back to CUDA 12.
Building every variant in one CI job ran the runner out of disk and cancelled the ROCm and Intel builds. Each variant now builds in its own job, on its own runner.

Status and limitations #

winnow is at v0.6.6 (June 2026), AGPL-3.0, on Python 3.13, and needs Immich 1.106+ and Frigate 0.16+. It has just over 100 unit tests, 33 of them for the selection algorithms, and GitHub Actions builds all four images and publishes them to GHCR. The NVIDIA path is the one with production hours on it.

The limits:

The README labels it early development. Quality replacement and stale-mapping cleanup remove images from Frigate and aren’t yet proven at scale.
Selection embeds Immich’s preview thumbnails, while the uploaded crop comes from the original. They usually agree, but heavy preview compression could shift an embedding.
Frigate’s score compares against the mean of a person’s training embeddings. A set split between frontal shots and profiles has its mean in between, so candidates from either side look more novel than they are.

What I’d do differently #

In v0.5.0 I moved the tracker from two JSON files to SQLite. What followed, per the changelog: 17 bug-fix releases in two days, for data-loss risks in the migration layer, primary-key conflicts and retry storms on a full disk. v0.6.0 went back to JSON. Atomic writes followed a day later, and main now adds a file lock for overlapping runs. For two files of asset IDs, a database bought nothing that os.replace and flock don’t. Next time I’d name what new storage solves before paying for a migration.

Next, if Frigate’s API ever exposes per-image embeddings, winnow can measure diversity against the whole training set, hand-added images included, instead of treating the score as a proxy.

Links #

Repository: github.com/sudolulo/winnow
Setup, troubleshooting and FAQ: the wiki
Upstream project: ds-sebastian/if_curator