rdma rxe not creating ib device

[jrambo316]

Hello,
I am trying to test SoftRocev2 - software based rdma over ethernet on a Talos bare metal cluster.

minimum requirements seem to be:
rdma_rxe kernel module
rdma command
ibverbs-utils package

Successful test with ubuntu vm:
sudo modprobe rdma_rxe
sudo apt-get install ibverbs-utils
note: rdma command available by default
sudo rdma link add rxe0 type rxe netdev eth0
rdma link show -->success link active and up
ibv_devices -->success shows rxe0 ibdevice
ibv_devinfo -->success shows details

Talos os:
rdma_rxe loaded via kernel module machine config
kubectl debug -n kube-system -it --image debian node/kube1-wk1 --profile=sysadmin
apt-get install ibverbs-utils
note: rdma tool not available
apt-get install iproute2 -->rdma available in /bin
rdma link add rxe0 type rxe netdev <long talos name>
rdma link show -->success link active and up
ibv_devices -->failure shows zero ibdevices
ibv_devinfo -->failure shows zero device details

In my multiple attempts, I tried some other ib tools where I found some failure messages related to not being able to open umad ports but I think that might be a dead end. I tried on Talos with ib_umad kernel module on and off with no difference. Also, I attempted chown 666 to the umad0 and umad1 files that were created which made no difference. The successful ubuntu vm test did not require the ib_umad module.

Is there perhaps something in Talos that just blocks the ibdevice creation or detection?

Regards,
James

[smira]

Talos should have first class support to make it real, but inside kubectl debug container /dev is not really host /dev (the /host/dev is), so you might need some nsenter fun to make it work

[jrambo316]

Thanks for your reply!

Update:
I finally was able to get the ibv_devices and ibv_devinfo to show the devices in Talos.
Steps:
kubectl debug
apt-get install iproute2
apt-get install ibverbs-utils
rdma link add ...
rdma link show ...
ibv_devices --> fail

exit the debug shell

kubectl debug ... again
apt-get install iproute2
apt-get install ibverbs-utils
rdma link show ... --> success link still there
ibv_devices --> success device rxe0 shows
ibv_devices --> success device details show
rdma link add rxe1 type rxe netdev <second talos interface>
rdma link show --> success both links show
ibv_devices -->mixed success ONLY the rxe0 shows

exit the debug shell

kubectl debug ... again
apt-get install iproute2
apt-get install ibverbs-utils
rdma link show ... --> success link still there
ibv_devices --> success device rxe0 AND rxe1 show
ibv_devices --> success device details FOR BOTH devices show

Any ideas on what is happening?

Regards,
James

[jrambo316]

FYI...

I was able to perform rdma perftest tests using method above between two worker nodes. However, I am still puzzled why I have to logout and kubectl debug again to views ib devices successfully.

Regards,
James

[smira]

See above, /dev inside the container is not host's /dev. It might be repopulated as you re-login.