VPP main thread stuck at session_wrk_handle_evts_main_rpc() when working with nginx through VCL

[WangZongyuanSimon]

DUT setup like this:

               (eth1 + bvi if)  bridge1    
                       |
http client -------- VPP     +     VCL + nginx(reverse proxy) 
                       |
               (bvi if + eth2) bridge2
                       | ---------------------------------------  http backend server

VPP starts with two workers, session configure:
session {
enable
use-app-socket-api
event-queue-length 100000
preallocated-sessions 100000
evt_qs_memfd_seg
}

start nginx with LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG
VCL configure:
vcl {
heapsize 512M
#use-mq-eventfd 1
app-socket-api /var/run/vpp/app_ns_sockets/default
segment-size 536870912
add-segment-size 536870912
event-queue-size 500000
rx-fifo-size 65536
tx-fifo-size 65536
}

http client can get/post backend server through the reverse proxy Nginx with curl in the first, which means the http traffic path is OK.
But when I use wrk to do performance test, with http traffic forwarding, and then reload nginx by kill -HUP NGINX-PID, then VPP will stuck at session_wrk_handle_evts_main_rpc().

Tried VPP with version 22.10/23.10/25.10, all with the same problem.
enable/disable use-mq-eventfd with no improvement.
If I increase event-queue-length from 8192 to 100000/500000, then symptoms have eased, what I mean is,
event-queue-length = 8192, the first time reload nginx, stuck happens.
event-queue-length = 100000, the first time reload nginx, VPP can work and vppctl show sessions can be displayed. but when I reload nginx the second time(30s later), stuck happens.

This is easy to reproduce, seems a problem about syncing from worker to main thread about the tcp connections, see the attached for more details.

more_info.txt

[florincoras]

A few quick comments/questions:

• no need for evt_qs_memfd_seg in startup.conf
• event-queue-length does need to be configured appropriately for the use case. If it's too small it'll probably lead to session layer concluding that application is not draining fast enough and marking it as congested. Check that with sh app mq
• am I understanding correctly that the issue only happens when nginx is hup-ed? If yes, that might be a bug in the code that tries to synchronize events when one of them needs to be handled on main thread (typically a listen). It can also be a side effect due to the load, i.e., session layer tries to synchronize current batch of events by executing them on main thread but constant stream of connects due to the test, i.e., due to wrk, will probably force all control events to happen through main instead of first worker.

Regarding the last, if you stop test for some seconds, does http client work again?

[WangZongyuanSimon]

Thanks Florin for your kind reply!

(1) sure, I removed the "evt_qs_memfd_seg" config and test again, the same result.

(2) "show app mq" displays that the event queue should be enough
[Ctrl0] [Q:0/100000] [R0:0/100000] [R1:0/50000]
[Ctrl1] [Q:0/100000] [R0:0/100000] [R1:0/50000]
[Ctrl2] [Q:0/100000] [R0:0/100000] [R1:0/50000]
[A0][0] [Q:0/0] [R0:0/0] [R1:0/16]
[A1][1] [Q:0/128] [R0:0/128] [R1:0/16]
[A2][2] [Q:0/128] [R0:0/128] [R1:0/16]
[A3][3] [Q:1/500000] [R0:0/500000] [R1:1/31250]
[A3][5] [Q:1221/500000] [R0:0/500000] [R1:1221/31250]

When I run kill -HUP to reload the nginx, the nginx worker process can be recovered after a few while, but VPP main thread will never recover, vppctl will always failed to connect. Then I tried with http client it says "Coundn't connect to server".

(3) Yes, issue only happens after nginx reload with traffic on, without traffic everything works fine. Since the VPP always stuck at session_wrk_handle_evts_main_rpc and never return back, it seems most like a bug in code about the session events synchronize.