Disaster recovery

Snapshots protect you from yourself. Replication protects you from a host. Neither protects you from losing the fleet — a snapshot on a dead pool and a replica on a dead rack are both gone.

DR Manager is the layer under that. It is a separate appliance holding encrypted restic repositories, deliberately outside the thing it protects, and it is the only part of the platform designed on the assumption that everything else is already on fire.

Deploy it somewhere the fleet is not. A DR Manager on a node it backs up is a backup of your infrastructure stored inside your infrastructure. Different site, different power, different failure domain — otherwise you have built a slower replica.

The idea worth understanding first

Ordinary backup systems keep long-lived credentials on every machine they back up. Steal one host and you can read, or delete, everything.

DR Manager inverts that. Every repository is closed by default. Before a backup, the host knocks: it authenticates with that resource’s own basic-auth credential and proves the exact name + UUID + repository binding it claims. Only then is the repository opened, only for the credential that opened it, and only for that transfer.

What comes back from a successful knock is the ingress URL, the canonical repository name, and the repository’s restic encryption password — meaning the host does not store the encryption password at all. It receives it per backup and forgets it.

The consequences are the point:

• A compromised host reaches exactly one repository — its own.
• Credentials do not have to be provisioned onto hosts out of band.
• A knock with a mismatched name or UUID is refused before any data is streamed.
• The restic ingress is append-only, so a host that is taken over cannot erase history.

Deploying it

DR Manager is a microVM running a compose stack, deployed exactly like Control Center — read that page first if you have not run one; the mechanics are identical and only the variables differ.

Four images, sharing one immutable tag: hoster-dr-backend, hoster-dr-frontend, hoster-dr-caddy and hoster-dr-tls-init. The registry is a variable, DR_REGISTRY; point it wherever yours are published.

./HosterDRManager/containers.sh both 20260801-230000

The build publishes that exact tag and never touches latest, which is the behaviour you want from the system you will be relying on during an incident.

Deploy it on an encrypted dataset

Not a recommendation — a requirement, and the reason is specific:

data/database contains restic repository passwords in plain text. It has to: the manager hands them to authenticated clients, which is what lets hosts hold no encryption keys. So the appliance’s own storage is the crown jewels.

• Losing that directory can make every repository permanently unrecoverable.
• Leaking it can make every repository decryptable.

Deploy the microVM only on an encrypted container dataset.

Protect and back up all of these, encrypted, and never inside the backup system they unlock:

Storing the only copy of the key database inside the backup system it unlocks is the mistake this table exists to prevent.

Configuration

cp example.env .env
chmod 600 .env
openssl rand -hex 32     # DR_JWT_SECRET
openssl rand -hex 32     # TRUSTED_PROXY_SECRET — a different one
docker compose config --quiet
docker compose pull
docker compose up -d

Both public URLs must be the externally reachable origin, port included. Hosts are told to dial what is in these variables; a value that only works from inside the microVM produces backups that fail at the knock.

The microVM template must have hoster-compose.service enabled and /data mounted shared — the FUSE mounts the backend creates for remote filesystems depend on that propagation:

systemctl is-enabled hoster-compose.service
findmnt -o TARGET,PROPAGATION /data

If the template predates that unit, install the bundled hoster-dr-manager.service instead — but never enable both on one microVM.

Publishing it

Only TCP 443 needs forwarding. The management UI, the API and the restic ingress all share that one origin, with restic routed under /restic:

hoster firewall bridge add-dnat hoster-internal \
  --target-microvm dr-manager \
  --external-address 192.0.2.10 \
  --external-port 2443 \
  --internal-port 443 \
  --protocol tcp \
  --description "DR Manager HTTPS and restic ingress"

Then verify from outside:

docker compose ps
curl -kfsS https://192.0.2.10:2443/health >/dev/null

The certificate is self-signed and persisted across restarts. Hoster nodes must trust it before you rely on this in production — or terminate a CA-issued certificate at the public endpoint instead.

Turning it on for a guest

DR is configured per guest, and the same shape works for both types:

hoster vm dr-config web-01 \
  --enable \
  --url https://192.0.2.10:2443 \
  --username web-01-dr \
  --password '<the credential issued by DR Manager>' \
  --schedule daily

hoster microvm dr-config registry --enable --url … --username … --password … --schedule weekly

hoster vm dr-config web-01 --disable

--schedule takes daily, weekly, monthly or yearly; empty means daily. --enable and --disable are mutually exclusive.

Note the credential is per resource, not per host. That is the whole security model: one guest’s credential opens one guest’s repository.

Scheduling is automatic

Enabling DR is enough. The scheduler builds a job from each guest’s own DR configuration — every non-backup resource with DR enabled gets one — so there is no generate-dr-jobs step to remember and nothing to re-run after adding a guest.

Jobs run one at a time, FIFO, so a fleet of guests does not put a fleet of restic processes on one host at once.

Replicas are skipped. A backup copy is not the machine, and backing it up would store the same data twice under two identities.

On demand

hoster vm dr-backup web-01
hoster vm dr-backup web-01 --json
hoster microvm dr-backup registry

Run one by hand before any change you are not sure about, and after enabling DR, so you find out about a bad URL or a rejected credential now rather than during an incident.

What a backup actually does

Worth knowing, because it explains the disk usage and the leftovers:

1. Refuses immediately if the resource is a replica, DR is not enabled, or the URL and credentials are missing.
2. Knocks, and gets back the ingress URL, repository name and encryption password.
3. Takes a snapshot of type dr.
4. Clones that snapshot to a temporary dataset named DR__<name>, and marks the clone as a backup so nothing in the fleet mistakes it for the guest.
5. Streams the clone’s contents to restic through the opened ingress.
6. Destroys the clone and its snapshot — deferred, so cleanup happens even on failure.

The guest is never stopped: restic reads a frozen clone, not the live dataset.

An interrupted run leaves a DR__<name> clone behind, and the next run resumes it. Rather than starting over, the backup reuses the existing clone and lets restic continue incrementally. So a DR__ dataset is not necessarily garbage — a stale one from a run that will never resume is, and it holds space until something removes it. If cleanup fails, the failure is written into the job log specifically so a leftover clone is never silent.

Restoring

A restore is authorised by a capability URL issued from DR Manager for one snapshot of one resource. It is short-lived, and deliberately not tied to a destination host, so an operator can paste it into whichever agent has the capacity to take the workload.

hoster dr restore 'https://192.0.2.10:2443/restore/<grant-id>/<token>' \
  --parent-dataset rpool/hoster/vm-encrypted \
  --name web-01-dr \
  --bridge hoster-internal \
  --start

--parent-dataset is required — nothing guesses where a recovered guest should land.

The host claims the capability, restores the restic snapshot into a hidden temporary dataset, validates it, and makes it visible with one atomic rename. A restore in progress is invisible to inventory, so a half-restored guest never appears in hoster vm list and nothing starts it by accident. A failed restore cleans up after itself.

The defaults are clone-safe

This is the part to read before an incident. By default a restored guest is treated as a copy, not as a resurrection: it gets a new UUID, a new IP and MAC, regenerated cloud-init, and its DR schedule, replication settings, external disks and PCI assignments are dropped.

That is the right default. Restoring is often a test, or a rebuild alongside a limping original — and a restore that came back claiming the original’s identity would collide with it, or worse, start writing into the original’s DR repository.

When you genuinely are recovering the machine, opt back in:

--preserve-identity is not “keep everything”. It does not imply --keep-external or --keep-pci — host-specific attachments stay disabled, because the hardware and the host paths a guest was using are exactly what is least likely to exist on the machine you are restoring onto. Add those two explicitly if the destination really does have them.

--bridge maps the restored interfaces onto a network that exists on this host, and --firmware-path / --kernel-path override microVM boot paths when the destination has them somewhere else. Getting those wrong is the common way a restore succeeds and then will not boot.

Restore drills

The one operational habit worth building: restore something on a schedule, into a throwaway name, with the default clone-safe flags, and start it.

hoster dr restore '<grant-url>' --parent-dataset rpool/hoster/vm-encrypted \
  --name restore-drill --start
hoster vm console restore-drill      # confirm it actually boots
hoster vm destroy restore-drill

The defaults make this safe to do against production repositories — the drill cannot collide with the running guest, because it is not pretending to be it.

When a backup fails

The failure is almost always at the knock, and the message says which:

A name or UUID mismatch usually means the guest was restored or rebuilt and now carries a new identity while DR Manager still expects the old one. That is the clone-safe default doing its job; re-issue the credential for the new identity.

Next

• Restoring a guest — the cheaper restore paths to try before this one.
• The scheduler — DR jobs are scheduler jobs; this is how you see whether one ran.
• MicroVMs — the guest type DR Manager runs on.
• Deploying Control Center — the same deployment pattern, documented step by step.