Troubleshooting a node

The other pages in this section assume the node is basically working. This one is for when it is not: the host itself, the generated configuration underneath every service, and the debris a crashed operation leaves behind.

Most of it lives under hoster debug, which is a worse name than it deserves — several of these are ordinary recovery tools rather than developer instrumentation.

Look at the host

hoster host info        # OS, hardware, system configuration
hoster host metrics     # live CPU, memory, disk and network
hoster host zfs         # pool status, dataset usage, health
hoster host resources   # all three in one report

hoster host resources is the one to capture when you are about to ask someone else for help — it is the other three concatenated, so it answers the first round of questions without a second round trip.

hoster datasets           # every ZFS dataset, VM and container
hoster datasets --json
hoster pci list           # host PCI devices and passthrough readiness
hoster pci list --json

hoster pci list reports passthrough readiness, not just presence — which device is bound to what, and whether it could be handed to a guest. Note that on the VM side this is currently informational: see PCI passthrough for why a VM cannot be given one from the CLI.

hoster datasets and hoster host datasets are the same command implemented twice. Two files, byte-identical apart from the variable name and the example text, both aliased to ds and dataset. Harmless today, except that a fix to one will not reach the other.

Regenerate the configuration underneath everything

Several services do not read Hoster’s configuration directly — they read files Hoster generates from it. When the two disagree, this is the command that resettles them:

hoster debug generate-config hoster_dns       # DNS records
hoster debug generate-config kea-dhcp-v4      # DHCP server config
hoster debug generate-config resolv-conf      # the host's own resolver
hoster debug generate-config rqlite           # rqlite runtime config
hoster debug generate-config host-config      # a default host_config.json
hoster debug generate-config network-config   # a default network_config.json

The first four regenerate from current state and are safe to re-run. The last two are different — they generate default files. On a configured node that is not a refresh, it is a reset, so treat host-config and network-config as bootstrap commands rather than repair ones.

These same files are regenerated automatically on every boot, which is why a reboot often “fixes” a config drift and why the honest fix is regenerating rather than rebooting. See Quick start, stage 2.

Then reload whatever consumes them:

hoster service reload  hoster_dns
hoster service restart hoster_dns
hoster service restart dhcp

hoster service only knows two services. Despite Short text reading “Restart a service using its name”, the parent takes no name argument — it accepts exactly zero args and prints help. Only the hardcoded subcommands work: restart dhcp, restart hoster_dns, and reload hoster_dns. There is no reload dhcp, and nothing for the agent, the scheduler, rqlite or Prometheus — those are systemctl on the host.

Clearing what a crash left behind

Two kinds of debris, both with the same shape: an operation died, its bookkeeping outlived it, and the next attempt trips over it.

Stale lock files

Hoster serialises the operations that must not overlap — replication, network interface changes, image downloads — with lock files in /opt/hoster/run/. A process that dies mid-operation leaves its lock behind, and the next attempt waits for it and then gives up.

The symptom is a bare operation timed out after a few seconds, on something that should not take a few seconds. The lock acquire retries about twenty times at 150ms before failing, so failure is quick and unexplained.

hoster debug lock-files clear-all

Make sure the operation really is dead first. Clearing the lock of a running replication is how you get two of them writing at once — which is exactly what the lock exists to prevent. Check with hoster scheduler jobs --running true and hoster debug replication-state before clearing.

Orphaned TAP interfaces

Each running guest gets a TAP interface on its bridge. A guest that was killed rather than stopped can leave one behind, which shows up as an interface on the bridge with no guest, and occasionally as a name collision when the guest starts again.

hoster debug tap-interfaces clear-by-vm <vmName>   # one guest's interfaces
hoster debug tap-interfaces clear-all              # every TAP interface

Prefer clear-by-vm. clear-all destroys the TAP interfaces of running guests too, which disconnects them from the network without stopping them — a worse problem than the one you started with. clear-all is for a node with no guests running.

Replication that will not settle

hoster debug replication-state   # local ZFS replication states
hoster debug replication-info    # local replication configuration

The pair answers the two halves separately: what is configured versus where the transfers actually got to. A replication that never runs is usually the first; one that runs and never completes is the second.

For scheduled replication, the scheduler is the better starting point — it has the error text.

Talking to QEMU directly

hoster debug qemu-monitor web-01 'info status'
hoster debug qemu-monitor web-01 'info block'

Both arguments are required: the guest name and one quoted command. This is the raw QEMU monitor, so what you can ask is whatever the guest’s QEMU supports, and the response is passed straight back.

This is the sharpest tool here. The monitor can stop, reset and reconfigure a running guest, and nothing validates the command. Use the info family freely; anything that changes state should go through the ordinary commands, which know about the configuration on disk. hoster vm stop --force already sends quit for you.

Is virtualisation actually available?

hoster debug kvm-ok

Reports the KVM device, whether it is accessible, the KVM API version and the CPU virtualisation flag. Worth knowing: this is Hoster’s own, and works on every distribution.

kvm-ok on its own is Ubuntu’s, from the cpu-checker package — not present on most distributions. If older material tells you to run kvm-ok, use hoster debug kvm-ok instead.

The usual cause of a failure here is virtualisation disabled in firmware, or a nested setup where the outer hypervisor is not exposing it. See Quick start for the nested case.

Small tools

hoster debug generate-mac              # a MAC in Hoster's range
hoster debug generate-vm-name          # a random test guest name
hoster debug prometheus dump-rqlite    # raw rqlite metrics
hoster version

generate-mac matters more than it looks: addresses generated this way sit in the range the platform expects, so a hand-built guest configuration does not collide with a generated one.

An order that works

When a node is misbehaving and you do not yet know why:

1. hoster host resources — is this a resource problem? A full pool or exhausted memory explains most “everything is slow” reports at once.
2. hoster network status — desired versus actual, with drift. See Networks.
3. hoster scheduler stats and jobs — is something failing repeatedly in the background? See The scheduler.
4. hoster firewall status and hoster firewall show — is the rule set what you think it is?
5. hoster debug replication-state — is a transfer wedged?
6. hoster debug lock-files clear-all — only after confirming nothing is running.
7. hoster debug generate-config … then reload — when a service disagrees with the configuration.
8. hoster vm console <name> — when the node is fine and one guest is not. See Getting into a guest.

The first four are read-only and cost nothing. Everything from 6 onward changes state, so work down the list rather than starting at the bottom.

Next

• The scheduler — where a background failure reports itself, with the error attached.
• Monitoring and alerts — how you find out about most of this before someone tells you.
• Getting into a guest — when the node is healthy and the guest is the problem.