Holden Salomon

A TrueNAS backup patch that rebuilds itself every boot and refuses to half-apply

An overlayfs patch that adds Backblaze B2 and S3 to TrueNAS's built-in restic backups, survives reboots and updates, and stands down when TrueNAS changes.

8 min read

TrueNAS SCALE’s built-in backup, TrueCloud Backup, wraps restic: it can snapshot a dataset with ZFS before each run, deduplicates, runs on a schedule and shows progress in the UI. It only backs up to Storj, and in 2026 Storj raised its TrueNAS tier from $5 to $50 a month. I already pay for Backblaze B2. truenas-truecloud-patch extends TrueCloud Backup to B2 and any S3-compatible provider, and optionally lets it snapshot datasets that have children, which stock TrueNAS refuses. It works by patching TrueNAS’s internal middleware, which has no stability contract, so most of the engineering is about what happens when the patch meets a TrueNAS it didn’t expect.

What it does #

Clone the repo onto a data pool and run install.sh as root. B2 and S3 credentials appear in TrueCloud Backup’s dropdown, and create_task.py verify confirms the patch is live. update.sh moves between release tags and rolls back in one command, and a TrueNAS alert says when a release is out. There is deliberately no auto-update: on a box that patches itself at boot, a bad commit detonates on the next reboot.

How it works #

A top-to-bottom boot sequence. middlewared starts with stock modules, pools import so the repository is reachable, and the PREINIT hook runs apply.sh. apply.sh asks whether this module's assumptions about middleware still hold. If not, the module is skipped and TrueNAS is left stock. If so, it mounts an overlay on the /usr directories with its upper layer in /run, appends patch blocks after a marker line, and schedules a deferred unit that waits for boot to settle, re-applies and verifies the patch on the live path, then restarts middlewared so the patched modules load. Every hour, an alert source checks whether the patch's stamps are present in the running process, and raises a CRITICAL alert if not.
The patch is rebuilt on every boot, and any step that can fail leaves TrueNAS stock rather than half-patched.

What had to change #

TrueNAS’s backend is a Python service, middlewared; its web UI is a compiled Angular bundle. Reading both turned up three gaps.

  • The UI filters the credential list through filterByProviders, set to ["STORJ_IX"]. The patch widens it to ["STORJ_IX","S3","B2"] in the compiled JavaScript, after backing up the original.
  • B2 credentials. Stock’s B2 remote class has no get_restic_config(), so a B2 task falls through to the base class’s NotImplementedError. The patch supplies one. Issue #1 showed that the credential layout differs between releases:
def _tc_get_restic_config(task):
    p = task["credentials"]["provider"]
    if not isinstance(p, dict):
        # TrueNAS <= 24.10: provider is the type string ("B2") and the
        # account/key live in the credential's attributes dict. 25.04+
        # moved them into a provider dict.
        p = task["credentials"]["attributes"]
    return "", {"B2_ACCOUNT_ID": p["account"], "B2_ACCOUNT_KEY": p["key"]}

B2RcloneRemote.get_restic_config = staticmethod(_tc_get_restic_config)
B2RcloneRemote.restic = True
  • The repository URL came out with a stray leading slash and a slash separator, where restic 0.16 wants b2:bucket:path. A wrapper rewrites it, and does nothing when the URL is already right.

Why overlayfs #

TrueNAS is an appliance: /usr is read-only and an update replaces it wholesale. So apply.sh mounts an overlay filesystem over the few middleware directories it touches. An overlay stacks a writable upper directory on a read-only lower one; reads fall through to the original unless the upper layer holds its own copy. Here the upper layer lives in /run, which is RAM-backed. The patched files exist only for the current boot, nothing under /usr is ever written, and undoing the patch is an unmount.

Persistence comes from the two things an update doesn’t replace: the repo lives on a data pool, and the boot hook is registered in TrueNAS’s configuration database as a PREINIT init script. The first boot after an update is just a boot: the overlay goes back up and the patch re-applies to whatever middleware is now installed.

Each patch is a Python block appended to a stock module after a marker line, so re-applying and reverting both start by truncating at the marker. CI compile()s every block, because a syntax error there would break the box at boot.

Staying safe across updates #

The patch writes down what it assumes about middleware: each function it wraps, whether it is async, and the parameters it passes. tools/compat.py checks that list twice. CI runs it daily against iXsystems’ unreleased branches and files a bug report while a breaking change is still in beta, and apply.sh runs it at every boot, refusing to apply any module whose assumptions fail. An unapplied module means stock TrueNAS: the feature is missing, but nothing is half-patched. The two modules also retire independently when TrueNAS ships either feature natively.

TrueNAS 26 put this to the test. It rewrote the cloud backup code from async to synchronous and deleted the ZFS methods the nested module called; either change alone would have broken backups. The obvious replacements, the public pool.dataset.query family, turned out to hide the datasets TrueNAS considers its own: 84 of 270 on a real pool, including live app data. Staging from that view would have silently left them all out. The module now follows one rule: read the truth from ZFS, make changes through middleware.

Patched on disk is not patched in the process #

My first model of the boot order was wrong. I assumed PREINIT scripts ran before middlewared; they are run by middlewared, after it has imported the stock modules, and it never re-imports. So B2 support silently reverted on every reboot, while verify read a status file that only knew what was on disk and said all was well. The fix was a deferred restart, whose first version fired two seconds into TrueNAS’s own boot work and killed the apps startup. That release took 54 apps down. wait_restart.sh now waits for systemd’s boot queue to drain and Docker’s state machine to settle, then restarts middlewared once.

On 2026-08-19 it failed in a new way. apply.sh mounted its overlay at 16:41:56 and logged every step OK. Four seconds later another boot hook ran systemd-sysext refresh, which unmerged and remerged /usr and took the overlay with it. The deferred restart loaded stock modules, and every B2 task failed for nineteen hours while apply.log said the patch was active.

Hook ordering can’t fix that, because middlewared itself merges a sysext over /usr at runtime when it configures NVIDIA support for Docker. So the restart script now re-applies and checks the live path right before it restarts. And “is the patch loaded?” is asked where it can be answered exactly: inside middlewared. The patch stamps the objects it replaces, and an hourly alert source looks for the stamps:

try:
    from middlewared.plugins.cloud_backup.restic import get_restic_config
except Exception:
    return False
if not getattr(get_restic_config, "_truecloud_patched", False):
    return False

A missing stamp raises a CRITICAL TrueNAS alert, and turning off update alerts doesn’t silence it.

A snapshot option that would have lied #

Stock TrueNAS refuses “Take Snapshot” on any dataset with children, which rules out every pool running Apps, since each app is its own dataset. I’d already been burned by a backup that stayed green for months while missing 622 GB, so the guard’s reason interested me more than its inconvenience. ZFS doesn’t expose child datasets through a parent’s .zfs/snapshot directory, so without the check restic walks a near-empty tree and reports SUCCESS. The guard is right. It has to be replaced with a real traversal, not removed.

So after the recursive snapshot, the patch bind-mounts every descendant’s own snapshot directory into a staging tree that mirrors the original layout, and points restic there. If any descendant can’t be staged, the backup fails, and a post-mount check asserts that every target is mounted and the root isn’t empty. The guard is relaxed last, so a partial apply leaves it in place. Datasets with no file tree to stage, like the legacy-mountpoint ones behind that 622 GB, are skipped and logged by name, not dropped silently.

Recursive snapshots exposed something stock never had to handle: its cleanup deletes only the parent snapshot, which on a 250-dataset pool orphaned 249 snapshots every successful run. The patch now owns the lifecycle, and ZFS pushed back. Reading a snapshot directory automounts it, by default for 300 seconds after the last access, so whatever restic read last is still busy when the sweep runs. Unmount-and-retry clears about 255 of 256; the rest is recorded in a sidecar file and reclaimed next run. The sidecar lives in /run, which a reboot wipes, so a garbage collector also finds leftovers by name.

On a 252-dataset pool, a scheduled backup built a 173-mount staging tree and finished in 18 minutes 14 seconds with zero orphaned snapshots. The same backup reading live files had stalled at 74% for over 12 hours.

The 55-day estimate #

Without a cache path on the task, TrueNAS runs restic with --no-cache, which re-reads repository metadata from the provider on every run. The first backup of a 564 GB dataset was estimated at 55 days. create_task.py gained --cache-path and warns when it’s missing.

Status and limitations #

  • v0.7.0 (July 2026) is the current stable release. v0.8.0, with the re-apply before restart and the in-process alert, is at its third release candidate.
  • Requires TrueNAS SCALE 24.10 or newer. The daily matrix passes 24.10, 25.04, 25.10 and 26.0.0-BETA.3, and flags iXsystems’ development branch for 27 as broken; I’m not chasing that until it settles.
  • Verified on real hardware running 25.10.4: three B2 tasks run end to end with no leftover snapshots or mounts, and a child-dataset file restored with a matching MD5. On a TrueNAS 26 beta VM: a 274-snapshot backup of a 292-dataset pool and a byte-identical restore from four levels deep.
  • About 350 tests. CI runs shellcheck, bash -n, ruff and pytest on Python 3.11 to 3.13, on both a self-hosted Gitea runner and GitHub.
  • It patches internal APIs, so “fail-safe” means the box stays up, not that backups keep running. 24.10 and 25.04 pass the static checks, but nobody has run a backup on them. Each boot costs one middlewared restart, and a backup fired in the short window before it fails once.

What I’d do differently #

On 2026-07-13 I cut twelve releases in one day, several fixing the one before, and with update alerts live each one interrupted every user. An alert people learn to dismiss is worse than none, because one day it carries a security fix. Now CI only publishes a stable tag on a commit a release candidate already points at, and candidates are invisible to update.sh and the alert. I’d build that gate before the alert, not after.