Skip to content

Podman in LXC: why containers lose their network after an SSH login

We run many services with rootful Podman in unprivileged LXC containers on Proxmox. For weeks the same error kept coming back: after a podman stop or a re-create via Ansible, a service was suddenly unreachable – “No route to host”, even locally.

The symptom

podman stop reported open container netns … No such file or directory. The container then started with a new IP, but netavark left the old port forwarding (DNAT rule) in place. Because the old rule matches first, traffic went to an IP that no longer existed.

The cause

Rootful Podman keeps its network namespaces under /run/user/0/netns. On the first SSH login as root, however, systemd-logind mounts a fresh tmpfs over /run/user/0 – and unmounts it, including its contents, on the last logout.

  • Namespaces created at boot sit underneath the tmpfs and are invisible in every SSH session.
  • Namespaces created during an SSH session are gone completely after logout.

In both cases the next podman stop cannot find its namespace, doesn’t clean up the network, and netavark leaves an orphaned rule behind. Ansible and CI runners log in via SSH – which is exactly why the error mostly hit automated workflows.

You can prove it harmlessly with a bind mount that looks underneath the tmpfs:

d=$(mktemp -d); mount --bind /run $d; ls $d/user/0/netns; umount $d

The fix

loginctl enable-linger root

With this, user-runtime-dir@0 runs permanently from boot and /run/user/0 is never unmounted and remounted again. In addition, a drop-in for podman-restart.service with Wants= and After=user-runtime-dir@0.service makes the containers start only after it at boot. Then restart the LXC once.

We rolled this out as an Ansible task and tested it on every guest: stop/start via SSH, new IP, exactly one DNAT rule, service reachable internally and externally.

What else we learned

  • podman-compose down deletes and recreates the network and is particularly prone to this. To re-create a container, it’s better to remove just the container and run podman-compose up -d.
  • podman-compose up --no-deps in version 1.3 still creates all containers in the stack and fails if neighbours are running. Instead of trusting the exit code, we check whether the containers are actually running.
  • After every re-create of a container with an exposed port: check reachability, not just the status.