Running vma inside docker container

[Cha465]

Hi, is it viable to preload libvma inside docker container, in a loopback scenario (server and client run in the same container)? I preload libvma.so before executable, but the output of strace still shows sendmsg/recvmsg syscalls, and vma_stats finds no offloaded packets.

[igor-ivanov]

Hi @Cha465 the behavior is not container specific.
It is expected that scenario using tcp connection on the same server when both (server and client) processes under libvma whould work.

[Cha465]

Hi, I tried again with sockperf, specifying tcp connection:
LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf sr --tcp -i 0.0.0.0 // server
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13
VMA INFO: Cmd Line: sockperf sr --tcp -i 0.0.0.0
VMA INFO: Current Time: Mon Dec 25 09:31:13 2023
VMA INFO: Pid: 77119
VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0:
VMA INFO: Architecture: x86_64
VMA INFO: Node: e55bdebd6779
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA Spec Latency [VMA_SPEC]
VMA INFO: Log Level INFO [VMA_TRACELEVEL]
VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX]
VMA INFO: Tx QP WRE 256 [VMA_TX_WRE]
VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING]
VMA INFO: Rx QP WRE 256 [VMA_RX_WRE]
VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING]
VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL]
VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL]
VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX]
VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL]
VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE]
VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO]
VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS]
VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL]
VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE]
VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT]
VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC]
VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL]
VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY]
VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD]
VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY]
VMA INFO: Thread mode Single [VMA_THREAD_MODE]
VMA INFO: ---------------------------------------------------------------------------
sockperf: == version #3.7-no.git ==
sockperf: [SERVER] listen on:
[ 0] IP = 0.0.0.0 PORT = 11111 # TCP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: [tid 77119] using recvfrom() to block on socket(s)

LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf pp -i 0.0.0.0 --tcp -t60 // client
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13
VMA INFO: Cmd Line: sockperf pp -i 0.0.0.0 --tcp -t60
VMA INFO: Current Time: Mon Dec 25 09:31:31 2023
VMA INFO: Pid: 77125
VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0:
VMA INFO: Architecture: x86_64
VMA INFO: Node: e55bdebd6779
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA Spec Latency [VMA_SPEC]
VMA INFO: Log Level INFO [VMA_TRACELEVEL]
VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX]
VMA INFO: Tx QP WRE 256 [VMA_TX_WRE]
VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING]
VMA INFO: Rx QP WRE 256 [VMA_RX_WRE]
VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING]
VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL]
VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL]
VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX]
VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL]
VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE]
VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO]
VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS]
VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL]
VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE]
VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT]
VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC]
VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL]
VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY]
VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD]
VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY]
VMA INFO: Thread mode Single [VMA_THREAD_MODE]
VMA INFO: ---------------------------------------------------------------------------
sockperf: == version #3.7-no.git ==
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 0.0.0.0 PORT = 11111 # TCP
sockperf : Warmup stage ( sending a few du mmy mess ERSION: 9.7.1-0 Developm ent Snap shot bui sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: [Total Run] RunTime=60.000 sec; Warm up time=400 msec; SentMessages=4570324; ReceivedMessages=4570323
sockperf: ========= Printing statistics for Server No: 0
sockperf: [Valid Duration] RunTime=59.550 sec; SentMessages=4540388; ReceivedMessages=4540388
sockperf: ====> avg-latency=6.536 (std-dev=2.148)
sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
sockperf: Summary: Latency is 6.536 usec
sockperf: Total 4540388 observations; each percentile contains 45403.88 observations
sockperf: ---> observation = 2635.175
sockperf: ---> percentile 99.999 = 69.872
sockperf: ---> percentile 99.990 = 17.672
sockperf: ---> percentile 99.900 = 12.896
sockperf: ---> percentile 99.000 = 10.863
sockperf: ---> percentile 90.000 = 6.807
sockperf: ---> percentile 75.000 = 6.566
sockperf: ---> percentile 50.000 = 6.485
sockperf: ---> percentile 25.000 = 6.368
sockperf: ---> observation = 4.769

The performance is not ideal, and vma_stats -p <client_pid> is not showing a single line. What is the cause of this? Maybe the VMA ERROR must be fixed?

[igor-ivanov]

could you try to set specific ip instead of 0.0.0.0.

[Cha465]

127.0.0.1 or ip inside the docker container give the same result.

[igor-ivanov]

Of course, error should not be appeared.
Could you try on server first (w/o container) to separate potential issues.

Example on the same node:
Launch server

sudo env VMA_STATS_FILE=/tmp/libvma-sr.log LD_PRELOAD=libvma.so sockperf sr -i 1.1.1.10 --tcp

Statistic for server:

$ cat /tmp/libvma-sr.log
======================================================
        Fd=[24]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
- Foreign Address = [1.1.1.10:50613]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 1 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 2 / 313151 (100.00%) [miss/hit]
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
Rx poll: 3 / 0 (0.00%) [miss/hit]
======================================================

Launch client:

sudo env VMA_STATS_FILE=/tmp/libvma-cl.log LD_PRELOAD=libvma.so sockperf pp -i 1.1.1.10 --tcp

Statistic for client:

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:50613]
- Foreign Address = [1.1.1.10:11111]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 0 / 313152 (100.00%) [miss/hit]
======================================================

[Cha465]

I got these results:
server

cat /tmp/libvma-sr.log
======================================================
        Fd=[20]
- TCP, Blocked
Rx and Tx where not active
======================================================

client

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [0.0.0.0:48379]
Rx and Tx where not active
======================================================

[igor-ivanov]

Did you launch on Mellanox device? Did you see any error messages from VMA? Use direct ip as mentioned before.

[Cha465]

• Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.
• No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):

 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************

• Yes, I used direct ip.

[igor-ivanov]

• Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.

Just use ip that is set on Mellanox NIC

• No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):

 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************

It is strange. Did you use different users to set ulimit and launch application?

• Yes, I used direct ip.

[SC-z]

I used libvma when I was using the in-container NIC passthrough mode, and here are some of my configurations. Device mapping and ulimit memlock are critical , Hope we can help you.

docker run --rm -it \
    --name=rdma_test \
    --privileged \
    --net=none \
    --hostname=rdma_test \
    --device=/dev/infiniband/rdma_cm \
    --device=/dev/infiniband/uverbs2 \
    --device=/dev/net/tun \
    --privileged \
    --ulimit memlock=-1:-1 \
    -v /data/dk:/mnt \
    rdma-test:1212