RHEL 9 install issue

[PipeItToDevNull]

This seems different than #444 as far as I can see at least.

I am seeing issues with this on a fresh RHEL 9 install, running in a rootfull configuration.

sudo dnf install -y nomad nomad-driver-podman podman podman-plugins
sudo systemctl enable --now podman.socket

The nomad-driver-podman package creates the following link:

sudo ls -al /opt/nomad/data/plugins/
total 0
drwxr-xr-x. 2 root  root  33 Nov  4 21:10 .
drwxr-xr-x. 6 nomad nomad 90 Nov  4 20:51 ..
lrwxrwxrwx. 1 root  root  28 Nov  4 21:10 nomad-driver-podman -> /usr/bin/nomad-driver-podman

Using the following configuration for /etc/nomad.d/nomad.hcl

data_dir  = "/opt/nomad/data"
bind_addr = "0.0.0.0"

client {
  enabled = true
  servers = ["10.10.10.10"]
}

plugin "nomad-driver-podman" {
  config {
    socket_path = "unix://run/podman/podman.sock"
  }
}

When nomad starts the driver is not loaded

sudo nomad plugin status
Container Storage Interface
No CSI plugins

The logs are strange to my uninitiated eyes: `sudo

[]: ==> Nomad agent started! Log data will stream in below:
[]:     2025-11-04T22:22:52.061-0500 [INFO]  agent.plugin_loader: plugin process exited: plugin_dir=/opt/nomad/data/plugins plugin=/opt/nomad/data/plugins/nomad-driver-podman id=3282
[]:     2025-11-04T22:22:52.133-0500 [INFO]  agent.plugin_loader: plugin process exited: plugin_dir=/opt/nomad/data/plugins plugin=/opt/nomad/data/plugins/nomad-driver-podman id=3290
[]:     2025-11-04T22:22:52.135-0500 [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
[]:     2025-11-04T22:22:52.135-0500 [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
[]:     2025-11-04T22:22:52.135-0500 [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
[]:     2025-11-04T22:22:52.135-0500 [INFO]  agent: detected plugin: name=podman type=driver plugin_version=0.6.3
[]:     2025-11-04T22:22:52.135-0500 [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
[]:     2025-11-04T22:22:52.135-0500 [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
[]:     2025-11-04T22:22:52.136-0500 [INFO]  client: using state directory: state_dir=/opt/nomad/data/client
[]:     2025-11-04T22:22:52.136-0500 [INFO]  client: using alloc directory: alloc_dir=/opt/nomad/data/alloc
[]:     2025-11-04T22:22:52.136-0500 [INFO]  client: using dynamic ports: min=20000 max=32000 reserved=""
[]:     2025-11-04T22:22:52.183-0500 [WARN]  client.fingerprint_mgr.consul: failed to acquire consul self endpoint: cluster=default error="Get \"http://127.0.0.1:8500/v1/agent/self\": dial tcp 127.0.0.1:85>
[]:     2025-11-04T22:22:52.189-0500 [WARN]  client.fingerprint_mgr.landlock: failed to fingerprint kernel landlock feature: error="function not implemented"
[]:     2025-11-04T22:22:52.198-0500 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=enX0
[]:     2025-11-04T22:22:52.201-0500 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
[]:     2025-11-04T22:22:52.213-0500 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=enX0
[]:     2025-11-04T22:22:52.226-0500 [WARN]  client.fingerprint_mgr.cni_plugins: failed to read CNI plugins directory: cni_path=/opt/cni/bin error="open /opt/cni/bin: no such file or directory"
[]:     2025-11-04T22:23:00.234-0500 [INFO]  client.proclib.cg2: initializing nomad cgroups: cores=0-1
[]:     2025-11-04T22:23:00.234-0500 [INFO]  client.plugin: starting plugin manager: plugin-type=csi
[]:     2025-11-04T22:23:00.234-0500 [INFO]  client.plugin: starting plugin manager: plugin-type=driver
[]:     2025-11-04T22:23:00.235-0500 [INFO]  client.plugin: starting plugin manager: plugin-type=device
[]:     2025-11-04T22:23:00.235-0500 [INFO]  client.plugin: starting plugin manager: plugin-type=dynamic_host_volume
[]:     2025-11-04T22:23:00.238-0500 [ERROR] client.driver_mgr.docker: failed to list pause containers for recovery: driver=docker error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock.>
[]:     2025-11-04T22:23:01.053-0500 [INFO]  client: started client: node_id=2a5c0d39-0689-2347-de57-1a5023928699
[]: Started Nomad.
[]:     2025-11-04T22:23:01.065-0500 [INFO]  client: node registration complete

[tgross]

Hi @PipeItToDevNull !

When nomad starts the driver is not loaded

sudo nomad plugin status
Container Storage Interface
No CSI plugins

Yeah, the nomad plugin status command only shows the status of CSI plugins, which run as their own Nomad jobs. So I wouldn't expect to see anything about the podman plugin there. (Also just FYI you shouldn't need to use sudo when using the CLI to communicate with Nomad. The CLI makes a HTTP request to whichever agent the NOMAD_ADDR env var points to.)

This log line:

[]: 2025-11-04T22:22:52.135-0500 [INFO] agent: detected plugin: name=podman type=driver plugin_version=0.6.3

shows that the Nomad node was able to detect the plugin correctly. The nomad node status -verbose $nodeid command will have a Drivers section that should show the plugin health and an Attributes section with fingerprinted attributes from the driver (among a lot of other data!). For example, on my host:

$ nomad node status -verbose -self
...
Drivers
Driver             Detected  Healthy  Message                             Time
docker             true      true     Healthy                             2025-11-05T16:53:08-05:00
exec               true      true     Healthy                             2025-11-05T16:53:08-05:00
exec2              true      true     Healthy                             2025-11-05T16:53:08-05:00
java               true      true     Healthy                             2025-11-05T16:53:08-05:00
mock_driver        true      true     Healthy                             2025-11-05T16:53:08-05:00
nomad-driver-virt  true      true     Healthy                             2025-11-05T16:53:08-05:00
podman             true      true     All Podman sockets are responding.  2025-11-05T16:53:08-05:00
qemu               true      true     Healthy                             2025-11-05T16:53:08-05:00
raw_exec           true      true     Healthy                             2025-11-05T16:53:08-05:00
...

Attributes
...
driver.podman                            = 1
driver.podman.appArmor                   = false
driver.podman.capabilities               = CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
driver.podman.cgroupVersion              = v2
driver.podman.defaultPodman              = true
driver.podman.health                     = healthy
driver.podman.ociRuntime                 = /usr/bin/crun
driver.podman.rootless                   = false
driver.podman.seccompEnabled             = true
driver.podman.selinuxEnabled             = true
driver.podman.socket                     = unix:///run/podman/podman.sock
driver.podman.socketName                 = default
driver.podman.version                    = 5.6.2

[PipeItToDevNull]

I see that now in the verbose output, thanks so much for the response and clarifications! This is my first dip into the Nomad ecosystem.