The scheduler

Everything that happens on a timer — snapshots, replication, DR backups, template syncing — is one scheduler on each host. Setting those schedules up is a different job from operating them, and this page is the second one: what is registered, what ran, what failed, and how to run something now instead of waiting.

One job at a time

The scheduler runs a single worker. Jobs are queued and executed FIFO, one at a time, however many are due at once.

That is deliberate. Snapshots, replication and DR all hit the same disks and the same uplink; running twenty at once would make each slower and the host less responsive than running them in sequence. The cost is that a long job delays the ones behind it, which is what queue_depth in the stats is for.

Is it healthy?

hoster scheduler stats
hoster scheduler stats --json

last_reload_at is the one people need and do not expect: if you edited a schedule file and the count did not move, the scheduler has not picked it up yet.

What is scheduled

hoster scheduler schedule               # every configured entry
hoster scheduler schedule snapshot
hoster scheduler schedule replication
hoster scheduler schedule templates

This is the configuration view — the entries that exist and when they are set to run. Use it to answer “should this have run at all”, before asking why it did not.

What actually ran

hoster scheduler jobs                   # recent history, newest first
hoster scheduler jobs --limit 200
hoster scheduler jobs --running true    # only what is executing now
hoster scheduler jobs --running false
hoster scheduler jobs snapshot
hoster scheduler jobs replication
hoster scheduler jobs templates

--limit defaults to 75. Each entry carries the job key and name, its type, its state, when it was scheduled, started and finished, how long it took, which attempt it was, and the error if it failed.

That last pair is the point of the command. “The backup did not happen” is usually either no entry (it was never scheduled — check schedule) or an entry with an error (it ran and failed, and the error says why).

Output formats

Every command here takes --output table|unix|json, with --json as a shortcut:

hoster scheduler jobs --json | jq '.jobs[] | select(.error != null)'

unix is the tab-separated form for grep and awk.

--socket is deprecated and ignored. Scheduler queries go over the local agent’s gRPC connection now. It is still accepted so old scripts do not break.

Running a job now

hoster scheduler debug run <job-key-or-name>
hoster scheduler debug run snap_hourly --category snapshot

This bypasses the queue and runs the job immediately with the local executor. It is the command for “I do not want to wait until 03:00 to find out whether this is fixed”.

Four things worth knowing before you use it:

1. The selector is an exact match on the job’s key or its name — not a substring and not a pattern. Get the exact value from hoster scheduler jobs or hoster scheduler schedule.
2. An ambiguous selector fails and lists the keys it matched, so you can pick one. --category narrows it.
3. It refuses if the job is already running or queued, rather than starting a second copy of something that is mid-flight.
4. It runs once. Retries are disabled for a debug run, so a failure is reported rather than quietly retried — which is what you want when you are testing a fix.

Definitions are reloaded before the job is looked up, so a schedule you just edited is picked up without restarting anything.

Where the schedules come from

Four files, and each has a generator:

Every generator takes the same two flags:

hoster scheduler generate-snapshot-jobs --dry-run    # print it, write nothing
hoster scheduler generate-snapshot-jobs --file /tmp/snapshot_schedule.json

--dry-run before writing is the habit worth having: these commands overwrite a live schedule file. The snapshot generator merges your custom entries with the defaults rather than discarding them, but seeing the result first costs nothing.

The replication and DR generators additionally take a cadence, which sets the cron expression on the entries they write — but not the same one:

The two sets barely overlap. Replication goes down from daily — every 15 minutes, hourly, daily. DR goes up from daily — daily, weekly, monthly, yearly. --daily is the only cadence both understand. So there is no built-in weekly replication and no built-in hourly DR backup.

And --cron is missing from exactly one of them. The standalone replication and DR generators both accept an explicit cron expression, which is the escape hatch when the presets do not fit. The cluster replication generator does not — so a cluster-wide replication schedule that is not 15-minutely, hourly or daily means editing the generated file by hand.

Which of these you actually need is covered in Snapshots, replication and the scheduler; DR is the exception, because enabling DR on a guest schedules itself and the generator is only an override.

Where the tooling is uneven

The scheduler runs more kinds of job than its CLI lets you look at.

jobs and schedule have subcommands for only three categories. snapshot, replication and templates. The scheduler itself also runs dr, template_replication, template_local_sync, microvm_templates, microvm_template_replication and microvm_template_local_sync — so DR backups are scheduled, executed and recorded, but there is no hoster scheduler jobs dr.

They are not invisible: the parent hoster scheduler jobs shows every category, and filtering by job_type in the JSON works:

Copy

hoster scheduler jobs --json | jq '.jobs[] | select(.job_type == "dr")'

stats has no DR counter. It reports replication_entries, snapshot_entries and template_entries — the same three — so the number of DR entries is not in the summary even though DR jobs are registered like any other.

debug run --category accepts more than its help says. The help text lists replication, snapshot, templates, template-replication, template-local-sync. The parser also accepts dr, all, microvm-templates, microvm-template-replication and microvm-template-local-sync, in both hyphenated and underscored spellings. So --category dr works; the help just does not mention it.

When something did not run

In order:

1. Is there an entry for it? hoster scheduler schedule — no entry means nothing was ever going to run.
2. Did the definitions reload? hoster scheduler stats → last_reload_at. A file edited after that time has not been picked up.
3. Did it run and fail? hoster scheduler jobs — the error is on the entry.
4. Is it stuck behind something? queue_depth plus --running true. One worker means one long job delays everything.
5. Does it work at all? hoster scheduler debug run <key> runs it now, once, with the error in front of you instead of in a log.
6. For DR specifically, the failure is usually the knock rather than the scheduler — see when a backup fails.

Next

• Snapshots, replication and the scheduler — setting up the schedules this page inspects.
• Disaster recovery — DR jobs are scheduler jobs, with their own failure modes.
• Restoring a guest — what the snapshots these jobs create are actually for.
• Monitoring and alerts — the other background system on every node, and how it tells you something is wrong.
• Building your own template — what the hourly template fan-out distributes.