Picking the few photos that teach a face recognizer the most

Frigate, the open-source NVR, can put names to faces, but only for people it has training images of. The people I most wanted it to know rarely walk past a camera, so it never collects enough varied shots of them. My Immich photo library already knows them by name, but uploading everything would be close to useless: a library is full of bursts, repeats and the same few angles, and a recognizer learns little from the tenth copy of one condition. winnow picks the handful of photos that differ most. By default it keeps at most five images per person, chosen from a pool of up to 3,000 candidates, and registers them through Frigate’s API.

What it does #

You point a container at Immich and Frigate and give it a cron schedule. Each run goes through every named person, chooses candidates, crops each face from the original photo, aligns it to the 112×112 layout ArcFace-style recognizers expect, and uploads it.

A tracker remembers which photos were uploaded or rejected, so none is processed twice, and maps each Frigate filename back to its Immich asset. That enforces the rule I care about most: winnow only deletes or replaces files it uploaded itself, never images someone curated by hand in Frigate’s UI.

It ships as four Docker images: NVIDIA CUDA (:latest), AMD ROCm (:rocm), Intel through OpenVINO (:intel), and a CPU image (:cpu) for both amd64 and arm64.

winnow began as a fork of Sebastian’s if_curator, an interactive tool that already had the embedding and clustering core. I rewrote it to run unattended in Docker and added near-duplicate removal, the whole Frigate side (delivery, the tracker, Frigate-scored replacement) and the multi-accelerator builds.

How it works #

The pool narrows in stages, cheapest first. Photos older than ten years, already uploaded or previously rejected drop out before any download. Preview thumbnails then pass a quality gate: Laplacian variance for blur (a sharp image has strong edges, so its second derivative varies a lot), near-identical colour channels for grayscale or infrared, mean brightness for exposure, a minimum face width of 90 pixels, and a minimum detection confidence of 0.7.

Survivors are cropped to the person’s face using Immich’s bounding box, so a group photo embeds the right person, and InsightFace’s ArcFace model turns each crop into a 512-dimension vector.

Why embeddings, and why these algorithms #

ArcFace is trained so that photos of the same person land close together and photos of different people land far apart. Within one person’s cloud of points, the distance that remains is mostly what a camera varies: pose, light, expression, glasses, age. So the cosine distance between two embeddings of one person measures how differently a recognizer sees the two photos. That is the right yardstick for diversity; timestamps and pixel differences are not.

Selection runs in three steps.

1. Near-duplicate removal. Sort by sharpness, then greedily drop any photo within 0.20 cosine distance of one already kept. Burst shots sit around 0.01 to 0.05. This keeps the sharpest frame of each burst and stops a dense knot of near-identical frames from pulling a cluster centre toward itself.
2. K-Medoids. Group the embeddings into k clusters and take each cluster’s medoid, the real photo nearest the middle of its group. (k-means would return an averaged vector that isn’t a photo at all.) Medoids guarantee one representative per natural “look”, such as the years with a beard and the years without. Initialisation is deterministic and the swap search uses a fixed seed, so reruns agree.
3. Farthest-point sampling (FPS). Fill any remaining slots by repeatedly taking the candidate whose distance to its nearest selected photo is largest. This greedily spreads the picks apart and reaches the outliers that clustering averages away. Candidates Immich detected with confidence below 0.85 get their distance boosted, by up to 30% at the 0.7 floor, because low confidence usually means a profile or a partly hidden face, which is what a doorway camera sees. In adaptive mode, selection stops once the best remaining candidate is closer than 20% of the median pairwise distance, so a person with little variety gets fewer images instead of padding.

One honest detail: k is capped at the budget, so with the default cap of five the medoids usually fill every slot. FPS does its real work when the cap is raised, or on small pools where k is limited to a third of the candidates.

Frigate judges novelty #

Before each upload, winnow sends the crop to Frigate’s recognize endpoint (about 200 ms). A high score means Frigate already recognizes this face easily, possibly thanks to hand-added images winnow can’t see. Below the cap, a candidate scoring higher than the most redundant image winnow uploaded is skipped as already covered. At the cap, a candidate replaces that image only if it scores lower. A first run has no model to score against, so scoring starts on the next one.

The GPU image that quietly ran on the CPU #

InsightFace runs on ONNX Runtime, which picks an execution provider (CUDA, ROCm, OpenVINO or CPU) at startup, and winnow falls back to the CPU when no GPU provider loads. That is friendly on a desktop and dangerous in a container, because the failure never looks like a GPU problem. In production it looked like a crash loop: InsightFace on the CPU, out of memory on large libraries.

The CUDA provider went missing in three separate ways:

• CUDA and cuDNN came from nvidia-* pip packages whose libraries the dynamic loader couldn’t see. Registering those directories with ldconfig at build time fixed it.
• insightface 1.0.1 started depending on the CPU onnxruntime package, which ships the same compiled module as the GPU build, so whichever installs last wins. Marking them as conflicts in uv didn’t stop uv sync installing both in parallel, and on GitHub Actions the CPU binary consistently won.
• onnxruntime-gpu 1.26 only loads its CUDA provider if three more NVIDIA pip packages are importable, and they weren’t declared.

The last fix is blunt on purpose. After the sync, the Dockerfile reinstalls the GPU build on its own, so there is no race left to lose:

elif [ "$VARIANT" = "gpu" ]; then \
    uv sync --frozen --no-dev --extra gpu && \
    ORT_GPU_VER=$(.venv/bin/python -c "import importlib.metadata; print(importlib.metadata.version('onnxruntime-gpu'))") && \
    uv pip install --python .venv/bin/python --no-deps --reinstall "onnxruntime-gpu==$ORT_GPU_VER"; \

It’s worth the trouble: on an RTX 2070 SUPER the bundled benchmark embeds a face in 12.8 ms, about eight times the CPU’s throughput. The memory blow-up had a second cause, too: every candidate thumbnail was held at once, about 2.4 GB for a person with 472 candidates. Batches of 32 now cap that at roughly 256 MB.

One Dockerfile, four accelerators, two architectures #

Each accelerator wants a different base: NVIDIA’s CUDA image, Ubuntu 22.04 for Intel (its GPU repository only publishes for that release), 26.04 for ROCm and CPU, and 24.04 for arm64. One Dockerfile picks its base from a two-part stage name (abridged):

ARG VARIANT=gpu

FROM --platform=$BUILDPLATFORM nvidia/cuda:12.9.2-cudnn-runtime-ubuntu24.04 AS base-amd64-gpu
FROM ubuntu:26.04 AS base-amd64-rocm
FROM ubuntu:22.04 AS base-amd64-intel
FROM ubuntu:26.04 AS base-amd64-cpu
FROM ubuntu:24.04 AS base-arm64-cpu

ARG TARGETARCH
FROM base-${TARGETARCH}-${VARIANT} AS build

Most of the bugs here were silent defaults:

• Docker’s automatic TARGETARCH argument is only in scope for FROM lines unless a stage re-declares it. Inside RUN it was simply empty, so a per-architecture branch never added the Python 3.13 repository.
• add-apt-repository starts a gpg-agent, which crashed under QEMU emulation in the arm64 build. Piping the key through gpg --dearmor instead failed worse: gpg exits 0 on bad input, so apt quietly skipped the repository. The fix was the original command with a throwaway GNUPGHOME.
• A CUDA 13.3 base demanded NVIDIA driver 575 or newer and rejected hosts on 570, the stable driver at the time. The GPU image went back to CUDA 12.
• Building every variant in one CI job ran the runner out of disk and cancelled the ROCm and Intel builds. Each variant now builds in its own job, on its own runner.

Status and limitations #

winnow is at v0.6.6 (June 2026), AGPL-3.0, on Python 3.13, and needs Immich 1.106+ and Frigate 0.16+. It has just over 100 unit tests, 33 of them for the selection algorithms, and GitHub Actions builds all four images and publishes them to GHCR. The NVIDIA path is the one with production hours on it.

The limits:

• The README labels it early development. Quality replacement and stale-mapping cleanup remove images from Frigate and aren’t yet proven at scale.
• Selection embeds Immich’s preview thumbnails, while the uploaded crop comes from the original. They usually agree, but heavy preview compression could shift an embedding.
• Frigate’s score compares against the mean of a person’s training embeddings. A set split between frontal shots and profiles has its mean in between, so candidates from either side look more novel than they are.

What I’d do differently #

In v0.5.0 I moved the tracker from two JSON files to SQLite. What followed, per the changelog: 17 bug-fix releases in two days, for data-loss risks in the migration layer, primary-key conflicts and retry storms on a full disk. v0.6.0 went back to JSON. Atomic writes followed a day later, and main now adds a file lock for overlapping runs. For two files of asset IDs, a database bought nothing that os.replace and flock don’t. Next time I’d name what new storage solves before paying for a migration.

Next, if Frigate’s API ever exposes per-image embeddings, winnow can measure diversity against the whole training set, hand-added images included, instead of treating the score as a proxy.

Links #

• Repository: github.com/sudolulo/winnow
• Setup, troubleshooting and FAQ: the wiki
• Upstream project: ds-sebastian/if_curator