Monitoring and alerts

Hoster does not ship a monitoring stack. It ships the configuration for one: a generated Prometheus scrape block, a large set of alert rules written against what the platform actually exposes, and an HTTP discovery endpoint so guests appear as targets without anyone maintaining a target list.

You bring Prometheus, Alertmanager and Grafana. Hoster tells them what to look at.

The one thing to know first

Only guests marked production, and currently online, are discovered. The generated scrape jobs are literally called hoster_vms_production_online and hoster_microvms_production_online.

So a guest is monitored because someone marked it:

hoster vm set-config production web-01 --set-true
hoster microvm set-config production registry --set-true

If a guest is missing from Prometheus, this is the first thing to check — before the scrape config, before the firewall, before anything. See the production flag, which is the same flag hoster vm start-all --production-only uses.

Generating the scrape config

hoster prometheus config          # the whole file, as an example
hoster prometheus config block    # only the Hoster-managed block
hoster prometheus config apply    # write it into prometheus.yml and reload

config prints a complete example configuration and is marked experimental — treat it as a starting point for a node with no Prometheus yet, not as something to regenerate over a running setup.

block is the one to reach for when you already have a Prometheus you care about. It prints just the Hoster-managed section, wrapped in markers:

##### HOSTER CONTROLLED BLOCK START #####
...
##### HOSTER CONTROLLED BLOCK END #####

The generated config embeds API discovery credentials. The discovery endpoint is authenticated, so the rendered output contains the credentials Prometheus will use. hoster prometheus config output is a secret in the same way hoster vm get-config is — do not paste it into a ticket.

What apply does

It rewrites only what is between the markers, leaving the rest of your prometheus.yml alone, then writes the alert rules and reloads the service.

hoster prometheus config apply
hoster prometheus config apply --check=false --reload=false
hoster prometheus config apply --file /etc/prometheus/prometheus.yml \
  --rules-file /etc/prometheus/rules/hoster_host_alerts.yml \
  --rules-available-dir /etc/prometheus/rules-available

Leave --check on. It runs promtool when it is available and refuses to replace a working configuration with a broken one — which matters because this command edits a live file rather than writing a new one.

The rules-available directory is the useful half of the rules story: alerts that are generated but not loaded, so you can move one into the enabled file when you decide you want it, rather than editing generated content.

What gets scraped

The managed block sets up jobs for the node and everything on it:

Guest jobs scrape every 30s; the default interval is 60s; disks are slower because SMART data does not change quickly and reading it is not free.

Every series is labelled with hoster_parent — the node it came from — so a single Prometheus watching several nodes can group and route by host without you relabelling anything.

Listing discovery output by hand

The same data the discovery endpoint returns, on the CLI:

hoster prometheus vms
hoster prometheus microvms
hoster prometheus vms --production-only --online-only
hoster prometheus vms --tag web
hoster prometheus vms --use-ips
hoster prometheus vms --json-pretty

--use-ips emits addresses rather than names, for a Prometheus that cannot resolve your guest DNS. Useful for confirming that a guest you expect to be monitored actually appears.

microvms has no --tag filter. hoster prometheus vms can filter by tag; the microVM equivalent takes --production-only, --online-only, --use-ips and --json-pretty but not --tag. Same command shape, one flag short.

The alert rules

Around eighty rules are generated, written against the metrics the platform actually produces. Broadly:

Two worth calling out because they catch problems before they are outages:

• HosterNodeDiskMayFillIn24Hours projects growth rather than waiting for a threshold. A disk at 60% and climbing fast is a more urgent page than one steady at 85%.
• HosterWireguardPeerHandshakeNeverEstablished distinguishes a mesh link that broke from one that never worked — a distinction that usually costs an hour to make by hand. See the mesh.

Windows guests get their own rules (HosterVmWindows*) covering services and disks with the metric names a Windows exporter produces, so a mixed fleet does not need two rule sets maintained by hand.

Alertmanager

hoster alertmanager config print   # the Hoster-managed config
hoster alertmanager config apply   # write it and reload
hoster alertmanager config apply --check=false --reload=false

print merges with an existing alertmanager.yml before printing, so you can see what applying would produce rather than what a blank node would get. --check validates with amtool where available; both default to on, and both are worth leaving on for the same reason as Prometheus.

The generated routing groups alerts by alertname, hoster_parent and instance, with a 30s group wait and a 5m group interval. Grouping by hoster_parent is what stops a node with a real problem from paging you once per guest.

Receivers are yours. Hoster generates the routing skeleton and a default receiver; where alerts actually go — email, chat, a webhook — is configuration it does not invent for you.

Grafana

Two dashboards ship in the repository root: grafana_dashboard.json for the platform and grafana_smartctl.json for disk health. Import them into your Grafana; they expect the job and label names the managed block produces, which is the reason to generate the scrape config rather than hand-roll an equivalent.

Standing it up

There is no single command for this, and the order matters:

1. Install Prometheus and Alertmanager on the node, or point an existing pair at it.
2. Mark the guests you care about as production.
3. hoster prometheus config apply — writes the scrape block and the rules, validates, reloads.
4. hoster alertmanager config apply — writes routing, reloads.
5. Add your receivers to alertmanager.yml, outside the managed section.
6. Import the Grafana dashboards.
7. Confirm targets are up, and that HosterPrometheusNotConnectedToAlertmanager is not firing — it is the rule that catches step 4 having silently not worked.

When a guest is not being monitored

1. Is it marked production? Nothing else matters until it is.
2. Is it online? Discovery covers production and online.
3. Does it appear in hoster prometheus vms? If not, the problem is on the Hoster side; if it does, the problem is Prometheus reaching it.
4. Can Prometheus resolve the name? Try --use-ips to rule out DNS.
5. Can it reach the guest’s exporter port? That is a firewall question.
6. Did the last apply land? The managed block is between the markers in prometheus.yml; if it is missing, apply failed or was run against a different --file.

Next

• Changing a guest after deploy — the production flag that decides what is monitored.
• The scheduler — the other thing running quietly on every node, with its own way of telling you it failed.
• Troubleshooting a node — what to do once an alert has told you something is wrong.
• The interface — Control Center’s own view of guest health, which is not a replacement for this.