OBS - Error mounting volume in OpenShift

[phalamaj-sendati]

Hello,

We are trying to use the obs-csi-driver on an OpenShift Cluster ( version 4.16.27 ) in Huawei Cloud and we have an error when mounting the PV in the nodes.

We were able to deploy the solution using the instructions in the repo ( https://github.com/huaweicloud/huaweicloud-csi-driver/blob/master/docs/obs/obs.md ) , with just two minor changes:

--- csi-obs-node.yaml	2025-01-15 22:23:20.875559848 +0800
+++ csi-obs-node-ocp.yaml	2025-01-23 01:01:41.248251660 +0800
@@ -20,6 +20,7 @@
       tolerations:
         - operator: "Exists"
       hostPID: true
+      hostNetwork: true
       containers:
         - name: liveness-probe
           volumeMounts:
@@ -29,7 +30,7 @@
           args:
             - --csi-address=/csi/csi.sock
             - --probe-timeout=3s
-            - --health-port=28889
+            - --health-port=28890
             - --v=5
         - name: node-driver-registrar
           image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0
@@ -63,7 +64,7 @@
               exec:
                 command: [ "/bin/sh", "-c", "/obs-csi/stop-server.sh" ]
           ports:
-            - containerPort: 28889
+            - containerPort: 28890
               name: healthz
               protocol: TCP
           livenessProbe:

• The change in the port is to avoid conflict with csi driver of SFS-turbo
• And the change "hostNetwork: true" is to make the pods able to see the metadata of the ECS instances at http://169.254.169.254.

After all of this we have the service class and the dinamic provisioning working good, but the problem raise when OpenShift tries to link the PV with the node to start the pod.

at the deployment of the pod we find the following events.

15m         Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/5385b829-fcf8-4d64-bda8-d80a896c28f2 -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}
15m         Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/63a07564-c231-4327-bcb6-c88c2be67b72 -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}
15m         Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/56233375-2a1e-4676-bf39-dd005e520929 -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}
13m         Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/0f8f5742-ead1-4040-86d6-c14de7dce86b -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}
104s        Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    (combined from similar events): MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/dd094d70-0e14-49c9-aa63-9fe8e6cdef1e -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}

The worker node has the following OS:

[core@host-192-168-0-197 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux CoreOS release 4.16
[core@host-192-168-0-197 ~]$ sudo find / -iname libcrypto.so.10
find: File system loop detected; ‘/sysroot/ostree/deploy/rhcos/deploy/3f79e452161ec82aadee7a6212efb89f9e520de82eeb9b6c24b7c8a6efadc3c3.1’ is part of the same file system loop as ‘/’.
/sysroot/ostree/deploy/rhcos/var/lib/containers/storage/overlay/89169d87dbe2b72ba42bfbb3579c957322baca28e03a1e558076542a1c1b2b4a/diff/usr/lib64/libcrypto.so.10
/sysroot/ostree/deploy/rhcos/var/lib/containers/storage/overlay/174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02/diff/usr/lib64/libcrypto.so.10
/sysroot/ostree/deploy/rhcos/var/lib/containers/storage/overlay/be1f7344519a2b9b41aa7154581b1b5503c2168abb3bc1edc99b656dca2426fe/diff/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/97356b4db3a0a55ab2528e77773b338614bfc7ca5515f467f18135bc5da28fb1/merged/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/89169d87dbe2b72ba42bfbb3579c957322baca28e03a1e558076542a1c1b2b4a/diff/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02/diff/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/91f312abe1f58eee97401eca3d34be22abf081f6f6fb29a8a77607ce42d262df/merged/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/be1f7344519a2b9b41aa7154581b1b5503c2168abb3bc1edc99b656dca2426fe/diff/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/8fd1a9c0df259a554ab0fa91c2b3200f845c280b35b5b1959febd00f9c24f35f/merged/usr/lib64/libcrypto.so.10
[core@host-192-168-0-197 ~]$ 

Attach you can find the definitions of the pod, the PV and the PVC in the Manifest.tgz ( tar gzipped ) file.

Best

[teoks-dev]

obsfs is added to your CoreOS? if not you might need to compile and add it - https://support.huaweicloud.com/intl/en-us/fstg-obs/obs_12_0001.html
If you are still interested to take this forward, feel free to contact me.

[chengxiangdong]

From the logs, it appears that the obsfs tool is missing on the node. The node is running the Red Hat OS, and obsfs does not provide a built package. Therefore, the CSI driver will not automatically install this tool. You need to compile the obsfs source code and install it manually on each node.

[phalamaj-sendati]

From the logs, it appears that the obsfs tool is missing on the node. The node is running the Red Hat OS, and obsfs does not provide a built package. Therefore, the CSI driver will not automatically install this tool. You need to compile the obsfs source code and install it manually on each node.

Hello team, as you can see in https://github.com/huaweicloud/huaweicloud-csi-driver/blob/master/docs/obs/obs.md ( the guide we use to deploy the OBS CSI driver ) the cluster should be running a daemonset ( one pod in each node ) with the OBS tools.

Something like this for a 5 nodes cluster.

# kubectl get all -A
NAMESPACE      NAME                                                 READY   STATUS    RESTARTS       AGE
...
kube-system    pod/csi-obs-controller-687dc77b4d-lm54p              5/5     Running   5 (167m ago)   3h30m
kube-system    pod/csi-obs-plugin-fvswm                             3/3     Running   3 (167m ago)   3h30m
kube-system    pod/csi-obs-plugin-l6d5b                             3/3     Running   3 (167m ago)   3h30m

To notice, Oopenshift use a appliance style of OS, so installing tools like in any other linux distro is not like copying some files.

[chengxiangdong]

Yes, the CSI driver runs as a daemon and installs obsfs on each node. This requires that the pre-built obsfs package is compatible with the node’s operating system.

The obsfs only provides installation package for Ubuntu 16 and CentOS 7. For other operating systems, compilation is required. For more details, please refer to: https://support.huaweicloud.com/intl/en-us/fstg-obs/obs_12_0001.html#section2.

In this case, we recommend manually compiling and installing the obsfs on the node.