We made the most important containers of a Proxmox cluster highly available – without shared storage, using only ZFS replication between two nodes. The principle is simple; the pitfalls were in the details.
The setup
- The containers live on a ZFS pool that exists on both nodes.
- Proxmox replicates them to the second node every 15 minutes (
pvesr), with a bandwidth limit because the network card shares its link with Corosync. - An HA rule with priorities makes sure the containers normally run on the main node, only move to the second node on failure, and move back afterwards (failback).
auto-rebalancestays off: on the second node the pool sits on an HDD, so containers should only run there in an emergency.
An important expectation to set: on failover, up to 15 minutes of changes can be lost. That is acceptable for websites, a password vault or a reverse proxy – not for a database with heavy write traffic.
Migrating while in production
To move containers onto the replicated pool we built a script: pre-copy while running, stop the container briefly, send the incremental delta, adjust the configuration, start, check ports, set up replication – and only add the HA rule after the first complete replication. Downtime per container was in the range of seconds.
Pitfall 1: a consumer SSD slows everything down
The target pool on the main node is a QLC SSD. During large copies its cache fills up quickly – and then ZFS blocks every container on that pool, not just the one being copied. The fix: throttle large copies (about 12 MB/s in our case) and run them overnight.
Pitfall 2: false alarms from random ports
The script compares listening ports before and after the restart. On one container containerd listened on 127.0.0.1 with a random port – a different one after every start. Result: a false alarm, although the service was running the whole time. Since then we only compare externally reachable ports and ignore 127.x and ::1.
Pitfall 3: the router session on the jump host
A Winbox session to a MikroTik router was still open on the jump host – in Safe Mode. During the migration the jump host was gone for 17 seconds, the session dropped, and RouterOS dutifully rolled back every change since Safe Mode was enabled. A freshly configured VPN access was gone.
Two consequences: leave Safe Mode as soon as you have verified a change, and ask before restarting a jump host whether a session is open there.
Conclusion
HA without shared storage works well with Proxmox, as long as you know the potential data loss and don’t overestimate the hardware. Most of the work is not the HA configuration itself but a clean, verifiable migration.