I’ve started to running more containers in my homelab with Podman, running inside AlamaLinux. Most of the times, things just worked. However today, I tried to setup similar containers on OpenSUSE Leap 15.16 host, and I came across an issue getting containers to communicate with one another. Well, spoiler alert, I found a solution, hence this blog post.

TL;DR - install cni-plugin-dnsname package from cni-plugin-dnsname build service. (No package from the official OpenSUSE repo).

Brief background:

  • Containers are run as rootless with an non-privileged user.
  • Containers share the same network.
  • But containers cannot ping each others.

Let’s see the configuration of the Podman network on OpenSUSE host:

❯ podman network inspect podman
[
     {
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "cni-podman0",
          "created": "2024-11-11T06:21:41.049400226Z",
          "subnets": [
               {
                    "subnet": "10.88.0.0/16",
                    "gateway": "10.88.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

Notice the value of dns_enabled, set to false.

Now, let’s see what the Podman network from the AlamaLinux host contains:

$ podman network inspect podman
[
     {
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "podman0",
          "created": "2024-11-11T06:20:17.781034096Z",
          "subnets": [
               {
                    "subnet": "10.88.0.0/16",
                    "gateway": "10.88.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

The value of dns_enabled is set to true as seen in the above output.

So, the fist thing I wanted to try was to set the dns_enabled: true to the podman network running on OpenSUSE. But how can this be done?

According to the help document, this option dns_enabled: true is the default behavior:

❯ podman version
Client:       Podman Engine
Version:      4.9.5
API Version:  4.9.5
Go Version:   go1.23.2
Built:        Tue Oct 22 12:00:00 2024
OS/Arch:      linux/amd64

❯ podman network create --help | grep 'disable-dns'
      --disable-dns             disable dns plugin

So, I kept searching online, and fortunately found the information from [1] very helpful.

Apparently, someone had had this issue, containers in the same network cannot talk to each others on OpenSUSE, before me since 2023. From [1], I further check both of my Linux hosts to compare other settings.

On OpenSUSE 15.6, the network backend used is cni, whereas on my AlamaLinux box it uses netavark.

❯ grep ^NAME /etc/os-release
NAME="openSUSE Leap"

❯ podman info --format {{.Host.NetworkBackend}}
cni

Here is the output from the podman info running on AlamaLinux 9:

$ grep -i ^name /etc/os-release
NAME="AlmaLinux"

$ podman info --format {{.Host.NetworkBackend}}
netavark

The solution shared from [1] was to change the network backend for Podman on OpenSUSE to using netavark. In theory, this should work. However, I wanted to find out if I can keep cni backend.

Here is what I did to get it working:

  1. Download the cni-plugin-dnsname package from Build Service:

curl -O -L https://download.opensuse.org/repositories/devel:/microos/15.6/x86_64/cni-plugin-dnsname-1.3.1-lp156.2.13.x86_64.rpm

  1. Install required dependency, dnsmasq and cni-plugin-dnsname.
❯ sudo zypper in dnsmasq

❯ sudo rpm -ivh cni-plugin-dnsname-1.3.1-lp156.2.13.x86_64.rpm
  1. Recreate the podman network on OpenSUSE and restart all my containers.
❯ podman network rm podman

❯ podman network create --disable podman

Note: --disable option is just to ignore the error should the podman network already exists.

Just like that, my containers could now communicate with each other. There are some good documents at [3],[4],[5] if you wanna read more about this topic.

References: