Issues with ceph-csi, Nomad, and Podman

[benemon]

Describe the bug

When attempting to create a Ceph RBD volume via the Ceph-CSI driver on Nomad, the node plugin fails with:

rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

This fails for both static and dynamic volume mounts.

For static mounts, the volume is created successfully but cannot be staged or mounted on the client.

For dynamic mounts, the volume fails to be created and fails with the above error

I was walking through this guide:

https://docs.ceph.com/en/latest/rbd/rbd-nomad/

But trying to use Podman instead of Docker.

• The environment is a set of 3 identical hosts.
• Nomad Server / Client, and Ceph are co-deployed on the hosts.
• The controller and nodes deploy succesfully.

$ podman version
Client:       Podman Engine
Version:      5.4.0
API Version:  5.4.0
Go Version:   go1.23.4 (Red Hat 1.23.4-1.el9)
Built:        Wed Feb 12 15:54:13 2025
OS/Arch:      linux/amd64

---

Environment details

• Image/version of Ceph CSI driver: quay.io/cephcsi/cephcsi:v3.14.2 (also tried v3.15.0)
• Helm chart version: N/A (using Nomad CSI integration, not Kubernetes)
• Kernel version: RHEL 9.6 - 5.14.0-570.12.1.el9_6.x86_64
• Mounter used:

[root@node1 ~]# lsmod | grep rbd
rbd                   155648  0
libceph               614400  1 rbd

• Orchestrator:

Nomad v1.10.5+ent
BuildDate 2025-09-10T12:00:47Z
Revision 2c21645c33d1447a55b84a235ba4dd9ba225501e

• Ceph cluster version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)

---

Steps to reproduce (dynamic)

1. Deploy Ceph cluster using cephadm.
2. Deploy Ceph-CSI RBD plugin as Nomad CSI plugin with pool and secrets.
3. Try and create a volume rbd-test.

---

Actual results

Dynamic

• Volume provisioning fails
• Controller logs show:

2025-10-01T20:13:21.684252331+01:00 stderr F I1001 19:13:21.684231       1 utils.go:265] ID: 2014 Req-ID: rbd-test GRPC call: /csi.v1.Controller/CreateVolume
2025-10-01T20:13:21.684280988+01:00 stderr F I1001 19:13:21.684277       1 utils.go:266] ID: 2014 Req-ID: rbd-test GRPC request: {"accessibility_requirements":{},"capacity_range":{"limit_bytes":2000000000,"required_bytes":2000000000},"name":"rbd-test","parameters":{"clusterID":"eb7e38f6-954d-11f0-957a-b0416f15f052","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}}]}
2025-10-01T20:13:21.684416476+01:00 stderr F I1001 19:13:21.684411       1 rbd_util.go:1411] ID: 2014 Req-ID: rbd-test setting disableInUseChecks: false image features: [layering] mounter: rbd
2025-10-01T20:13:21.691141742+01:00 stderr F E1001 19:13:21.691116       1 controllerserver.go:243] ID: 2014 Req-ID: rbd-test failed to connect to volume : failed to get connection: connecting failed: rados: ret=-22, Invalid argument
2025-10-01T20:13:21.691153007+01:00 stderr F E1001 19:13:21.691140       1 utils.go:270] ID: 2014 Req-ID: rbd-test GRPC error: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

Static

• Volume registration succeeds.
• During staging/mount, Nomad fails with:

failed to setup alloc: pre-run hook "csi_hook" failed: mounting volumes: node plugin returned an internal error

• logs show a similar error:

I1001 12:36:47.752182       1 utils.go:266] ID: 74 Req-ID: rbd/nomad-rbd-test GRPC request: {"staging_target_path":"/local/csi/staging/default/rbd-test/rw-file-system-single-node-writer","volume_capability":{"access_mode":{"mode":"SINGLE_NODE_WRITER"}}
E1001 12:36:47.753009       1 utils.go:291] ID: 74 Req-ID: rbd/nomad-rbd-test GRPC error: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

---

Expected behavior

The Ceph RBD volume should stage and mount successfully to the Nomad client in either static or dynamic workflows

---

Logs

• csi-rbdplugin logs: see above

---

Additional context

• Pool and user keyring are valid; rbd map on the same node works fine:

rbd ls -p <pool_name>
rbd info <pool_name>/<image_name>
sudo rbd map <pool_name>/<image_name> --id <ceph_user> --keyfile /etc/ceph/<user>.keyring

• Issue appears specific to Ceph-CSI when used in Nomad.

• Cluster health:

ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)
  cluster:
    id:     eb7e38f6-954d-11f0-957a-b0416f15f052
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node1,node2,node3 (age 7h)
    mgr: node1.pjfnba(active, since 7h), standbys: node3.bcisde, node2.scpkvw
    osd: 3 osds: 3 up (since 7h), 3 in (since 11d)
    rgw: 3 daemons active (3 hosts, 1 zones)

  data:
    pools:   8 pools, 137 pgs
    objects: 381 objects, 992 KiB
    usage:   217 MiB used, 2.7 TiB / 2.7 TiB avail
    pgs:     137 active+clean```

- Pools:

[ceph: root@node1 /]# rados lspools
.mgr
rbd
default.rgw.buckets.data
.rgw.root
default.rgw.buckets.index
lab.rgw.log
lab.rgw.control
lab.rgw.meta

- Controller:
```hcl
job "csi-rbd-controller" {
  datacenters = ["dc1"]
  node_pool   = "hci-pool"
  type        = "service"

  update {
    progress_deadline = "25m"
  }

  group "controller" {
    network {
      port "metrics" {
      }
    }

    task "csi-rbd-plugin-controller" {
      driver = "podman"

      template {
        data = <<EOF
[{
  "clusterID": "eb7e38f6-954d-11f0-957a-b0416f15f052",
  "monitors": [
    "node1",
    "node2",
    "node3"
  ]
}]
EOF
        destination = "local/config.json"
        change_mode = "restart"
      }

      config {
        image        = "quay.io/cephcsi/cephcsi:v3.14.2"
        force_pull   = false
        network_mode = "host"

        volumes = [
          "local/config.json:/etc/ceph-csi-config/config.json:ro",
          "/home/nomad/csi/keys:/tmp/csi/keys:Z"
        ]

        args = [
          "--type=rbd",
          "--controllerserver=true",
          "--drivername=rbd.csi.ceph.com",
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--instanceid=${node.unique.name}-controller",
          "--pidlimit=-1",
          "--logtostderr=true",
          "--v=5",
          "--metricsport=${NOMAD_PORT_metrics}"
        ]
        privileged = true
      }

      kill_timeout = "20m"

      csi_plugin {
        id        = "rbd.csi.ceph.com"
        type      = "controller"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 500
        memory = 512
      }

      service {
        name     = "ceph-csi-controller"
        provider = "nomad"
        port     = "metrics"
        tags     = ["prometheus"]
      }
    }
  }
}

job "csi-rbd-nodes" {
  datacenters = ["dc1"]
  node_pool   = "hci-pool"
  type        = "system"

  update {
    progress_deadline = "25m"
  }

  group "nodes" {
    task "csi-rbd-plugin-node" {
      driver = "podman"

      template {
        data = <<EOF
[{
  "clusterID": "eb7e38f6-954d-11f0-957a-b0416f15f052",
  "monitors": [
    "node1",
    "node2",
    "node3"
  ]
}]
EOF
        destination = "local/config.json"
        change_mode = "restart"
      }

      config {
        image        = "quay.io/cephcsi/cephcsi:v3.14.2"
        force_pull   = false
        network_mode = "host"

        volumes = [
          "local/config.json:/etc/ceph-csi-config/config.json:ro",
          "/home/nomad/csi/keys:/tmp/csi/keys:Z",
          "/lib/modules/${attr.kernel.version}:/lib/modules/${attr.kernel.version}:ro"
        ]

        args = [
          "--type=rbd",
          "--nodeserver=true",
          "--drivername=rbd.csi.ceph.com",
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--instanceid=${node.unique.name}-node",
          "--pidlimit=-1",
          "--logtostderr=true",
          "--v=5"
        ]
        privileged = true
      }

      kill_timeout = "20m"

      csi_plugin {
        id        = "rbd.csi.ceph.com"
        type      = "node"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 500
        memory = 512
      }

      service {
        name     = "ceph-csi-node"
        provider = "nomad"
      }
    }
  }
}

NOTE: have also tried IPs for mons. Same response.

• Volume:

id           = "rbd-test"
name         = "rbd-test"
type         = "csi"
plugin_id    = "rbd.csi.ceph.com"
capacity_min = "2G"
capacity_max = "2G"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

secrets {
  userID  = "nomad-csi"
  userKey = "my-key"
}

parameters {
  clusterID     = "eb7e38f6-954d-11f0-957a-b0416f15f052"
  pool          = "rbd"
  imageFeatures = "layering"
}

mount_options {
  fs_type = "ext4"       
}

I hope this is a case of PEBKAC, but I'm struggling to find reference for folks doing a modern Noamd version, with Ceph-CSI and Podman!

[nixpanic]

The RADOS error -22 suggests that Ceph-CSI can not connect to the Ceph cluster. Because this happens already at the Controller/CreateVolume procedure, there is no need to check the nodeplugin/NodeStageVolume call yet.

Error -22 is Invalid argument. I am not sure what argument could be wrong though...

The only reference I found is in #2254. Can you check that your userKey (and maybe userID too) is base64 encoded in the secrets? It isn't the raw value as what Ceph commands read from the keyfile/keyring.

[benemon]

Thanks for the response @nixpanic!

As you correctly highlight, it's the lack of explanation as to what the invalid argument is that has me stumped. I've been trying to search through the stack for a deeper level of insight into the problem but to no avail. Is there any way to enable debug output on the controller to perhaps see what's going on? I tried increasing the -v verbosity but it didn't seem to have an effect.

Anyway, as to your suggestion - if I fire up the cephadm shell and run:

[ceph: root@node1 /]# ceph auth get client.nomad-csi
[client.nomad-csi]
	key = AQBXItFoPHqiERAARKsetctcetcetcetc==
	caps mgr = "profile rbd pool=rbd"
	caps mon = "profile rbd"
	caps osd = "profile rbd pool=rbd"

I get a key which looks like it's already encoded - as per the (admittedly quite old guidance in the docs now) I've been using that as is for userKey, alongside nomad-csi as my userID e.g.

secrets {
  userID  = "nomad-csi"
  userKey = "AQBXItFoPHqiERAARKsetctcetcetcetc=="
}

As per your suggestion I've tried various permutations of b64 encoding the userID and the userKey and it always gives me the same -22 result:

Error creating volume: Unexpected response code: 500 (rpc error: 1 error occurred:
	* controller create volume: rpc error: controller create volume: CSI.ControllerCreateVolume: controller plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument)

What's baffling to me is that I can jump onto the controller and manually execute rbd commands with the same configuration and I can see images get created e.g.:

[admin@node1 nomad.d]$ nomad alloc exec -task csi-rbd-plugin-controller 76055e45 /bin/bash
[root@node2 /]# USER='client.nomad-csi'
[root@node2 /]# POOL='rbd'
[root@node2 /]# MONS='node1,node2,node3'
[root@node2 /]# KEY='AQBXItFoPHqiERAARKsetctcetcetcetc=='
IMG="csi-probe-$(date +%s)"
printf '%s' "$KEY" > /tmp/ceph.key
[root@node2 /]# # create (size is MiB), then verify via rbd and rados
CEPH_ARGS="--debug_ms=1 --debug_rados=5 --debug_rbd=5" \
rbd -n "$USER" --keyfile=/tmp/ceph.key -m "$MONS" \
  create -p "$POOL" "$IMG" --size 16 --image-feature layering; echo "create rc=$?"

rbd   -n "$USER" --keyfile=/tmp/ceph.key -m "$MONS" ls   -p "$POOL" --format json
rbd   -n "$USER" --keyfile=/tmp/ceph.key -m "$MONS" info -p "$POOL" "$IMG" || echo "rbd info: not found"
rados -n "$USER" --keyfile=/tmp/ceph.key -m "$MONS" -p "$POOL" ls | grep -E "rbd_header|$IMG" || true
2025-10-02T20:09:02.020+0000 7fcdca778e40  1  Processor -- start
2025-10-02T20:09:02.020+0000 7fcdca778e40  1 --  start start
2025-10-02T20:09:02.021+0000 7fcdca778e40  1 --2-  >> v2:172.16.100.20:3300/0 conn(0x55f1ad61e0c0 0x55f1ad620560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
2025-10-02T20:09:02.021+0000 7fcdca778e40  1 --  --> v1:172.16.100.10:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x55f1ad4102c0 con 0x55f1ad620ce0
2025-10-02T20:09:02.021+0000 7fcdca778e40  1 --  --> v1:172.16.100.30:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x55f1ad3fa740 con 0x55f1ad61a200
2025-10-02T20:09:02.021+0000 7fcdca778e40  1 --  --> v2:172.16.100.20:3300/0 -- mon_getmap magic: 0 -- 0x55f1ad47ac80 con 0x55f1ad61e0c0
2025-10-02T20:09:02.021+0000 7fcdc8a43640  1 --2-  >> v2:172.16.100.20:3300/0 conn(0x55f1ad61e0c0 0x55f1ad620560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0
2025-10-02T20:09:02.021+0000 7fcdc8a43640  1 --2-  >> v2:172.16.100.20:3300/0 conn(0x55f1ad61e0c0 0x55f1ad620560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.16.100.20:3300/0 says I am v2:172.16.100.20:48236/0 (socket says 172.16.100.20:48236)
2025-10-02T20:09:02.021+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3991600366 learned_addr learned my addr 172.16.100.20:0/3991600366 (peer_addr_for_me v2:172.16.100.20:0/0)
2025-10-02T20:09:02.021+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3991600366 >> v1:172.16.100.30:6789/0 conn(0x55f1ad61a200 legacy=0x55f1ad61a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down
2025-10-02T20:09:02.021+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3991600366 >> v1:172.16.100.10:6789/0 conn(0x55f1ad620ce0 legacy=0x55f1ad6231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down
2025-10-02T20:09:02.021+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3991600366 --> v2:172.16.100.20:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1ad626be0 con 0x55f1ad61e0c0
2025-10-02T20:09:02.023+0000 7fcdc8a43640  1 --2- 172.16.100.20:0/3991600366 >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 0x55f1ad620560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcdc000a660 tx=0x7fcdc0058a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1220fcf01ef8419f server_cookie=0 in_seq=0 out_seq=0
2025-10-02T20:09:02.023+0000 7fcdb7fff640  1 -- 172.16.100.20:0/3991600366 <== mon.1 v2:172.16.100.20:3300/0 1 ==== mon_map magic: 0 ==== 449+0+0 (secure 0 0 0) 0x7fcdc006d070 con 0x55f1ad61e0c0
2025-10-02T20:09:02.023+0000 7fcdb7fff640  1 -- 172.16.100.20:0/3991600366 <== mon.1 v2:172.16.100.20:3300/0 2 ==== config(3 keys) ==== 180+0+0 (secure 0 0 0) 0x7fcdc0068440 con 0x55f1ad61e0c0
2025-10-02T20:09:02.023+0000 7fcdb7fff640  1 -- 172.16.100.20:0/3991600366 <== mon.1 v2:172.16.100.20:3300/0 3 ==== mon_map magic: 0 ==== 449+0+0 (secure 0 0 0) 0x7fcdc0068860 con 0x55f1ad61e0c0
2025-10-02T20:09:02.023+0000 7fcdca778e40  1 -- 172.16.100.20:0/3991600366 >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 msgr2=0x55f1ad620560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2025-10-02T20:09:02.023+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3991600366 >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 0x55f1ad620560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcdc000a660 tx=0x7fcdc0058a90 comp rx=0 tx=0).stop
2025-10-02T20:09:02.023+0000 7fcdca778e40  1 -- 172.16.100.20:0/3991600366 shutdown_connections
2025-10-02T20:09:02.023+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3991600366 >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 0x55f1ad620560 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop
2025-10-02T20:09:02.023+0000 7fcdca778e40  1 -- 172.16.100.20:0/3991600366 >> 172.16.100.20:0/3991600366 conn(0x55f1ad615450 msgr2=0x55f1ad6178d0 unknown :-1 s=STATE_NONE l=0).mark_down
2025-10-02T20:09:02.023+0000 7fcdca778e40  1 -- 172.16.100.20:0/3991600366 shutdown_connections
2025-10-02T20:09:02.023+0000 7fcdca778e40  1 -- 172.16.100.20:0/3991600366 wait complete.
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 librados: starting msgr at
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 librados: starting objecter
2025-10-02T20:09:02.024+0000 7fcdca778e40  1  Processor -- start
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 --  start start
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 librados: setting wanted keys
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 librados: calling monclient init
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 --2-  >> [v2:172.16.100.30:3300/0,v1:172.16.100.30:6789/0] conn(0x55f1ad61a200 0x55f1ad629390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 --2-  >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 0x55f1ad62bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 --2-  >> [v2:172.16.100.10:3300/0,v1:172.16.100.10:6789/0] conn(0x55f1ad620ce0 0x55f1ad62e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 --  --> [v2:172.16.100.10:3300/0,v1:172.16.100.10:6789/0] -- mon_getmap magic: 0 -- 0x55f1ad476de0 con 0x55f1ad620ce0
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 --  --> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] -- mon_getmap magic: 0 -- 0x55f1ad41f3c0 con 0x55f1ad61e0c0
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 --  --> [v2:172.16.100.30:3300/0,v1:172.16.100.30:6789/0] -- mon_getmap magic: 0 -- 0x55f1ad624f70 con 0x55f1ad61a200
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 --2-  >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 0x55f1ad62bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 --2-  >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 0x55f1ad62bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.16.100.20:3300/0 says I am v2:172.16.100.20:48250/0 (socket says 172.16.100.20:48250)
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3442032461 learned_addr learned my addr 172.16.100.20:0/3442032461 (peer_addr_for_me v2:172.16.100.20:0/0)
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3442032461 >> [v2:172.16.100.30:3300/0,v1:172.16.100.30:6789/0] conn(0x55f1ad61a200 msgr2=0x55f1ad629390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.30:3300/0,v1:172.16.100.30:6789/0] conn(0x55f1ad61a200 0x55f1ad629390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:3300/0,v1:172.16.100.10:6789/0] conn(0x55f1ad620ce0 msgr2=0x55f1ad62e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2025-10-02T20:09:02.024+0000 7fcdca4cd640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:3300/0,v1:172.16.100.10:6789/0] conn(0x55f1ad620ce0 0x55f1ad62e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:3300/0,v1:172.16.100.10:6789/0] conn(0x55f1ad620ce0 0x55f1ad62e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1ad62eca0 con 0x55f1ad61e0c0
2025-10-02T20:09:02.024+0000 7fcdc8a43640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 0x55f1ad62bb40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0058fa0 tx=0x7fcdc0068e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2025-10-02T20:09:02.024+0000 7fcdb5ffb640  1 -- 172.16.100.20:0/3442032461 <== mon.1 v2:172.16.100.20:3300/0 1 ==== mon_map magic: 0 ==== 449+0+0 (secure 0 0 0) 0x7fcdc006d040 con 0x55f1ad61e0c0
2025-10-02T20:09:02.024+0000 7fcdb5ffb640  1 -- 172.16.100.20:0/3442032461 <== mon.1 v2:172.16.100.20:3300/0 2 ==== config(3 keys) ==== 180+0+0 (secure 0 0 0) 0x7fcdc00737c0 con 0x55f1ad61e0c0
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f1ad637940 con 0x55f1ad61e0c0
2025-10-02T20:09:02.024+0000 7fcdb5ffb640  1 -- 172.16.100.20:0/3442032461 <== mon.1 v2:172.16.100.20:3300/0 3 ==== mon_map magic: 0 ==== 449+0+0 (secure 0 0 0) 0x7fcdc0073be0 con 0x55f1ad61e0c0
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1ad637f10 con 0x55f1ad61e0c0
2025-10-02T20:09:02.024+0000 7fcdca778e40  1 librados: init done
2025-10-02T20:09:02.026+0000 7fcdb5ffb640  1 -- 172.16.100.20:0/3442032461 <== mon.1 v2:172.16.100.20:3300/0 4 ==== mgrmap(e 44) ==== 149903+0+0 (secure 0 0 0) 0x7fcdc007d450 con 0x55f1ad61e0c0
2025-10-02T20:09:02.027+0000 7fcdb5ffb640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6808/4280754441,v1:172.16.100.10:6809/4280754441] conn(0x7fcda80af8d0 0x7fcda80b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
2025-10-02T20:09:02.027+0000 7fcdb5ffb640  1 -- 172.16.100.20:0/3442032461 <== mon.1 v2:172.16.100.20:3300/0 5 ==== osd_map(967..967 src has 258..967) ==== 5650+0+0 (secure 0 0 0) 0x7fcdc01364e0 con 0x55f1ad61e0c0
2025-10-02T20:09:02.027+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6800/1091071277,v1:172.16.100.10:6801/1091071277] conn(0x55f1ad638ea0 0x55f1ad63b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
2025-10-02T20:09:02.027+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.10:6800/1091071277,v1:172.16.100.10:6801/1091071277] -- osd_op(unknown.0.0:1 2.3d 2:bed6a8b9:::csi-probe-1759435737.rbd:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e967) -- 0x55f1ad63b7d0 con 0x55f1ad638ea0
2025-10-02T20:09:02.027+0000 7fcdc9244640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6808/4280754441,v1:172.16.100.10:6809/4280754441] conn(0x7fcda80af8d0 0x7fcda80b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0
2025-10-02T20:09:02.027+0000 7fcdca4cd640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6800/1091071277,v1:172.16.100.10:6801/1091071277] conn(0x55f1ad638ea0 0x55f1ad63b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0
2025-10-02T20:09:02.028+0000 7fcdc9244640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6808/4280754441,v1:172.16.100.10:6809/4280754441] conn(0x7fcda80af8d0 0x7fcda80b1d90 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00a3b0 tx=0x7fcdbc005870 comp rx=0 tx=0).ready entity=mgr.74109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2025-10-02T20:09:02.028+0000 7fcdca4cd640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6800/1091071277,v1:172.16.100.10:6801/1091071277] conn(0x55f1ad638ea0 0x55f1ad63b290 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2025-10-02T20:09:02.028+0000 7fcdca4cd640  1 -- 172.16.100.20:0/3442032461 <== osd.0 v2:172.16.100.10:6800/1091071277 1 ==== osd_op_reply(1 csi-probe-1759435737.rbd [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 168+0+0 (crc 0 0 0) 0x7fcdb8009560 con 0x55f1ad638ea0
2025-10-02T20:09:02.028+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.20:6800/3862426723,v1:172.16.100.20:6801/3862426723] conn(0x55f1ad63c1f0 0x55f1ad63e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
2025-10-02T20:09:02.028+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.20:6800/3862426723,v1:172.16.100.20:6801/3862426723] -- osd_op(unknown.0.0:2 2.3c 2:3f35cc86:::rbd_id.csi-probe-1759435737:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e967) -- 0x55f1ad63ed10 con 0x55f1ad63c1f0
2025-10-02T20:09:02.028+0000 7fcdc9244640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.20:6800/3862426723,v1:172.16.100.20:6801/3862426723] conn(0x55f1ad63c1f0 0x55f1ad63e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0
2025-10-02T20:09:02.028+0000 7fcdc9244640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.20:6800/3862426723,v1:172.16.100.20:6801/3862426723] conn(0x55f1ad63c1f0 0x55f1ad63e5e0 crc :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2025-10-02T20:09:02.029+0000 7fcdc9244640  1 -- 172.16.100.20:0/3442032461 <== osd.1 v2:172.16.100.20:6800/3862426723 1 ==== osd_op_reply(2 rbd_id.csi-probe-1759435737 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 171+0+0 (crc 0 0 0) 0x7fcdbc00e070 con 0x55f1ad63c1f0
2025-10-02T20:09:02.030+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.30:6800/2247583985,v1:172.16.100.30:6801/2247583985] conn(0x55f1ad690690 0x55f1ad692b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
2025-10-02T20:09:02.030+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.30:6800/2247583985,v1:172.16.100.30:6801/2247583985] -- osd_op(unknown.0.0:3 2.3a 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e967) -- 0x55f1ad693080 con 0x55f1ad690690
2025-10-02T20:09:02.031+0000 7fcdc8a43640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.30:6800/2247583985,v1:172.16.100.30:6801/2247583985] conn(0x55f1ad690690 0x55f1ad692b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0
2025-10-02T20:09:02.032+0000 7fcdc8a43640  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.30:6800/2247583985,v1:172.16.100.30:6801/2247583985] conn(0x55f1ad690690 0x55f1ad692b20 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2025-10-02T20:09:02.032+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3442032461 <== osd.2 v2:172.16.100.30:6800/2247583985 1 ==== osd_op_reply(3 rbd_info [call out=4b] v0'0 uv2 ondisk = 0) ==== 152+0+4 (crc 0 0 0) 0x7fcdc007a020 con 0x55f1ad690690
2025-10-02T20:09:02.032+0000 7fcdca778e40  5 librbd::image::ValidatePoolRequest: read_rbd_info:
2025-10-02T20:09:02.032+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.30:6800/2247583985,v1:172.16.100.30:6801/2247583985] -- osd_op(unknown.0.0:4 2.3a 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e967) -- 0x55f1ad68f350 con 0x55f1ad690690
2025-10-02T20:09:02.033+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3442032461 <== osd.2 v2:172.16.100.30:6800/2247583985 2 ==== osd_op_reply(4 rbd_info [read 0~19 out=19b] v0'0 uv2 ondisk = 0) ==== 152+0+19 (crc 0 0 0) 0x7fcdc007a020 con 0x55f1ad690690
2025-10-02T20:09:02.033+0000 7fcdb77fe640  5 librbd::image::ValidatePoolRequest: handle_read_rbd_info: r=0
2025-10-02T20:09:02.033+0000 7fcdb77fe640  5 librbd::image::ValidatePoolRequest: finish: r=0
2025-10-02T20:09:02.033+0000 7fcdb77fe640  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.30:6800/2247583985,v1:172.16.100.30:6801/2247583985] -- osd_op(unknown.0.0:5 2.1c 2:3831950c:::rbd_directory:head [call rbd.dir_add_image in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e967) -- 0x7fcd9c008560 con 0x55f1ad690690
2025-10-02T20:09:02.118+0000 7fcdc8a43640  1 -- 172.16.100.20:0/3442032461 <== osd.2 v2:172.16.100.30:6800/2247583985 3 ==== osd_op_reply(5 rbd_directory [call] v967'17 uv17 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcdc007a020 con 0x55f1ad690690
2025-10-02T20:09:02.118+0000 7fcdb6ffd640  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.20:6800/3862426723,v1:172.16.100.20:6801/3862426723] -- osd_op(unknown.0.0:6 2.3c 2:3f35cc86:::rbd_id.csi-probe-1759435737:head [create,call rbd.set_id in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e967) -- 0x7fcd98003370 con 0x55f1ad63c1f0
2025-10-02T20:09:02.194+0000 7fcdc9244640  1 -- 172.16.100.20:0/3442032461 <== osd.1 v2:172.16.100.20:6800/3862426723 2 ==== osd_op_reply(6 rbd_id.csi-probe-1759435737 [create,call] v967'9 uv9 ondisk = 0) ==== 213+0+0 (crc 0 0 0) 0x7fcdbc00e070 con 0x55f1ad63c1f0
2025-10-02T20:09:02.194+0000 7fcdb77fe640  1 -- 172.16.100.20:0/3442032461 --> [v2:172.16.100.10:6800/1091071277,v1:172.16.100.10:6801/1091071277] -- osd_op(unknown.0.0:7 2.19 2:9a5a7c33:::rbd_header.12235521b2131:head [create,call rbd.create in=60b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e967) -- 0x7fcd9c004170 con 0x55f1ad638ea0
2025-10-02T20:09:02.200+0000 7fcdca4cd640  1 -- 172.16.100.20:0/3442032461 <== osd.0 v2:172.16.100.10:6800/1091071277 2 ==== osd_op_reply(7 rbd_header.12235521b2131 [create,call] v967'7 uv7 ondisk = 0) ==== 210+0+0 (crc 0 0 0) 0x7fcdb8009560 con 0x55f1ad638ea0
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6800/1091071277,v1:172.16.100.10:6801/1091071277] conn(0x55f1ad638ea0 msgr2=0x55f1ad63b290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6800/1091071277,v1:172.16.100.10:6801/1091071277] conn(0x55f1ad638ea0 0x55f1ad63b290 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 >> [v2:172.16.100.20:6800/3862426723,v1:172.16.100.20:6801/3862426723] conn(0x55f1ad63c1f0 msgr2=0x55f1ad63e5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.20:6800/3862426723,v1:172.16.100.20:6801/3862426723] conn(0x55f1ad63c1f0 0x55f1ad63e5e0 crc :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 >> [v2:172.16.100.30:6800/2247583985,v1:172.16.100.30:6801/2247583985] conn(0x55f1ad690690 msgr2=0x55f1ad692b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.30:6800/2247583985,v1:172.16.100.30:6801/2247583985] conn(0x55f1ad690690 0x55f1ad692b20 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6808/4280754441,v1:172.16.100.10:6809/4280754441] conn(0x7fcda80af8d0 msgr2=0x7fcda80b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.10:6808/4280754441,v1:172.16.100.10:6809/4280754441] conn(0x7fcda80af8d0 0x7fcda80b1d90 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00a3b0 tx=0x7fcdbc005870 comp rx=0 tx=0).stop
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 msgr2=0x55f1ad62bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 --2- 172.16.100.20:0/3442032461 >> [v2:172.16.100.20:3300/0,v1:172.16.100.20:6789/0] conn(0x55f1ad61e0c0 0x55f1ad62bb40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0058fa0 tx=0x7fcdc0068e60 comp rx=0 tx=0).stop
2025-10-02T20:09:02.200+0000 7fcdca4cd640  1 -- 172.16.100.20:0/3442032461 reap_dead start
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 shutdown_connections
2025-10-02T20:09:02.200+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 >> 172.16.100.20:0/3442032461 conn(0x55f1ad615450 msgr2=0x55f1ad58ac70 unknown :-1 s=STATE_NONE l=0).mark_down
2025-10-02T20:09:02.201+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 shutdown_connections
2025-10-02T20:09:02.201+0000 7fcdca778e40  1 -- 172.16.100.20:0/3442032461 wait complete.
2025-10-02T20:09:02.201+0000 7fcdca778e40  1 librados: shutdown
create rc=0
["csi-probe-1759435737"]
rbd image 'csi-probe-1759435737':
	size 16 MiB in 4 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 12235521b2131
	block_name_prefix: rbd_data.12235521b2131
	format: 2
	features: layering
	op_features:
	flags:
	create_timestamp: Thu Oct  2 20:09:02 2025
	access_timestamp: Thu Oct  2 20:09:02 2025
	modify_timestamp: Thu Oct  2 20:09:02 2025
rbd_id.csi-probe-1759435737
rbd_header.12235521b2131

So in terms of raw connectivity, it looks OK.

[Madhu-1]

@benemon please see #5330 this can help you to enable rbd/rados debug logs and see what exactly happening. As @nixpanic mentioned this is mostly related to the user/key. try using other user like admin and see if that works.

[benemon]

Hi @Madhu-1 @nixpanic

It looks like this is where it gets interesting. With a modicum of debug in the controller, I now see the following:

2025-10-03T12:02:00.548361837+01:00 stderr F I1003 11:02:00.548193       1 utils.go:265] ID: 10 Req-ID: rbd-test GRPC call: /csi.v1.Controller/CreateVolume
2025-10-03T12:02:00.548388461+01:00 stderr F I1003 11:02:00.548382       1 utils.go:266] ID: 10 Req-ID: rbd-test GRPC request: {"accessibility_requirements":{},"capacity_range":{"limit_bytes":2000000000,"required_bytes":2000000000},"name":"rbd-test","parameters":{"clusterID":"eb7e38f6-954d-11f0-957a-b0416f15f052","imageFeatures":"layering","pool":"rbd"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}}]}
2025-10-03T12:02:00.548608366+01:00 stderr F I1003 11:02:00.548552       1 rbd_util.go:1411] ID: 10 Req-ID: rbd-test setting disableInUseChecks: false image features: [layering] mounter: rbd
2025-10-03T12:02:00.551293473+01:00 stderr F 2025-10-03T11:02:00.549+0000 7f6a7d3bbac0 -1 auth: failed to decode key '[REDACTED]'
2025-10-03T12:02:00.554899435+01:00 stderr F 2025-10-03T11:02:00.553+0000 7f6a7d3bbac0 -1 auth: failed to decode key '[REDACTED]'
2025-10-03T12:02:00.555260503+01:00 stderr F 2025-10-03T11:02:00.553+0000 7f6a7d3bbac0 -1 auth: failed to decode key '[REDACTED]'
2025-10-03T12:02:00.555260503+01:00 stderr F 2025-10-03T11:02:00.553+0000 7f6a7d3bbac0 -1 auth: failed to decode key '[REDACTED]'
2025-10-03T12:02:00.555260503+01:00 stderr F 2025-10-03T11:02:00.553+0000 7f6a7d3bbac0 -1 monclient: keyring not found
2025-10-03T12:02:00.555357109+01:00 stderr F E1003 11:02:00.555347       1 controllerserver.go:243] ID: 10 Req-ID: rbd-test failed to connect to volume : failed to get connection: connecting failed: rados: ret=-22, Invalid argument
2025-10-03T12:02:00.555376882+01:00 stderr F E1003 11:02:00.555369       1 utils.go:270] ID: 10 Req-ID: rbd-test GRPC error: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

The auth: failed to decode key '[REDACTED]' / monclient: keyring not found items stick out to me.

If I look at the docs - https://docs.ceph.com/en/reef/rbd/rbd-nomad/ - I'm using the value of 'key' in the secret block for userKey, presented as-is.

The error there suggests that I should be using something else, or at least a different encoding here? Bearing in mind I've tried both admin and my specific nomad-csi user. Any thoughts?

[nixpanic]

Wondering how you did the base64 encoding. When I convert this

secrets {
  userID  = "nomad-csi"
  userKey = "AQBXItFoPHqiERAARKsetctcetcetcetc=="
}

to what I expect to be valid:

secrets {
  userID  = "bm9tYWQtY3Np"
  userKey = "QVFCWEl0Rm9QSHFpRVJBQVJLc2V0Y3RjZXRjZXRjZXRjPT0="
}

The conversion was done with these commands (note the -n to prevent the EOL):

$ echo -n 'nomad-csi' | base64 
bm9tYWQtY3Np
$ echo -n 'AQBXItFoPHqiERAARKsetctcetcetcetc==' | base64 
QVFCWEl0Rm9QSHFpRVJBQVJLc2V0Y3RjZXRjZXRjZXRjPT0=

The problem may well be something else, but I'd like your confirmation that you used a similar procedure.

[benemon]

I followed that process, with the exception of the -n on the echo to prevent the EOL. So I had, for example bm9tYWQtY3NpCg== as my representation of nomad-csi.

However, even though I've just reprocessed my userID and userKey - being sure to use -n - I still see the same -22 and output in the controller logs, unfortunately.

As before, this happens with both the admin userID / userKey and my own nomad-csi userID / userKey