Clustering

Clustering is not the same as having several machines. You can run twenty independent nodes and manage all of them from one Control Center without ever forming a cluster — Deploying Control Center works fine that way.

A cluster is for when hosts need to act on each other’s behalf.

What it turns on

What it costs

Three nodes. hoster cluster init takes exactly three addresses and will not bootstrap with fewer. That is not arbitrary: shared state needs a majority to agree, and two machines cannot form a majority when one of them is the problem.

A real network between them. The mesh and the shared state assume the nodes can reach each other and keep reaching each other. Latency you would not notice on a single host becomes visible in coordination.

If you have two machines and want off-site copies, you do not need a cluster — use standalone replication.

Forming one

Run this on the first node, listing all three. The first entry must be the node you are running it on:

hoster cluster init --nodes hoster0103,hoster0104,hoster0105

Entries can be a hostname, an address, or either with a port — all of these are valid:

hoster cluster init --nodes 192.168.193.203:2380,192.168.193.204:2380,192.168.193.205:2380
hoster cluster init --main-ip 192.168.193.203 --nodes hoster0103,hoster0104,hoster0105

--main-ip sets the address this node publishes as its own, when inferring it from routes would pick the wrong one — a machine with several interfaces, typically.

That command writes the first node’s cluster and agent configuration, generates the cluster CA and this node’s TLS material, and prints join tokens for the other two.

Then on each of the other nodes, with its token:

hoster cluster join <token>

The token carries the cluster CA, a one-time join secret and enough discovery information to find the others. The joining node generates its private key locally — it is never transmitted — requests a signed certificate over gRPC, writes its cluster and agent configuration, and refreshes the DNS records the nodes use to find each other.

Adding a fourth node later

The initial three are a bootstrap requirement, not a ceiling. To add another node, generate a token for it and join as before:

hoster cluster token hoster0106
# then, on hoster0106
hoster cluster join <token>

Looking at it

hoster cluster status              # leader, members, health
hoster cluster get nodes           # the cluster's own view of itself
hoster cluster get hosts           # the Hoster nodes in it
hoster cluster get vms             # every guest, wherever it lives
hoster cluster get networks
hoster cluster get wireguards
hoster cluster get latency         # per-host latency statistics

hoster cluster status takes --json, which is the one to script against.

get vms is the command that shows what a cluster is for: one list of every guest across every node, rather than logging into each machine and concatenating the answers.

The mesh, and cluster DNS

Forming a cluster builds a full WireGuard mesh between the nodes automatically. Each host keeps its own guest subnet — 10.0.1.0/24 on one, 10.0.2.0/24 on the next — and the mesh routes between them, so a guest on one host reaches a guest on another by address.

Cluster DNS is what makes that usable by name. It knows every guest across every node, and the records follow guests as they are created, moved or re-addressed. You can point your own resolvers at it and get name resolution for the whole fleet without writing a record by hand.

Both regenerate themselves when membership or configuration changes. There is no key exchange to perform and no config to edit.

→ DNS and the WireGuard mesh for where names come from, the --route-over-wg setting a network needs, and what to check when a name does not resolve.

When something is wrong

The debug commands answer “is it me, the network, or the other node?” without guessing:

hoster cluster debug liveness                  # gossip state as this node sees it (--all for every node)
hoster cluster debug rpc-ping [hostname]       # can this node reach the advertised gRPC endpoints?
hoster cluster debug wireguard-ping [hostname] # mesh reachability (--all for all-to-all)
hoster cluster debug wireguard-reload [host]   # regenerate and reload mesh configuration
hoster cluster debug dump                      # the cluster tables, as JSON

liveness --all and wireguard-ping --all are the two worth knowing: a problem that shows up from every node is not a problem with the node you are logged into.

Operational commands

These change cluster membership or state. They exist because real fleets have hardware that dies at inconvenient times.

reset-database is local and destructive. It removes this node’s cluster data files, cluster configuration and certificates. It is the way back from a broken join, not a way to fix a cluster.

Guests that can move

A guest deployed with --live-migration-enabled uses the cluster’s stored QEMU CPU model instead of passing the host’s CPU straight through. That is what lets it start on a different machine: a guest pinned to one host’s exact CPU features cannot be resumed on a host that lacks them.

Decide it at deploy time — see Your first VM — and work out the shared model with hoster cluster qemu-cpu calculate.

→ Live migration for the prerequisites, the preflight, and the list of things that rule a guest out.

Next

• Snapshots, replication and the scheduler — generate-replication-jobs reads cluster state, so this is where clustering starts paying for itself.
• Deploying Control Center — one interface over the whole cluster.