Concurrent support for tdvmcall interface : Recv getting called continuously even though VMM has no data

[saurabhbrcm]

We have the MigTD in working condition with a single connection at time but when adding support for concurrent connections using tdvmcall interface, I noticed the following:

• 1st WFR tdvmcall from MigTD : VMM does not have a vMotion request, so we return WFR tdvmcall with WFR_OP_NO_OP in response data->operation.

• VMotion request for 1st connection arrives : VMM posts the required data at the WFR physical address (0xa5f000) and then wakes up MigTD by posting interrupt. This connection is from destination VM first.

• The 2nd WFR arrives from MigTD : VMM does not have a 2nd connection request yet, so we return WFR tdvmcall with WFR_OP_NO_OP in response data->operation.

• 1st connection Send(STREAM_OP_REQUEST) tdvcmall is then made by MigTD: VMM gets Send call with op STREAM_OP_REQUEST.

• 1st connection Recv(STREAM_OP_RESPONSE) tdvcall : VMM injects a response to Send(STREAM_OP_REQUEST) with STREAM_OP_RESPONSE. The connection with VMM is now established. Connection 1 now goes in STREAM_OP_RW.

• 1st connection GetQuote call : GetQuote call from connection 1 succeeds.

• 1st connection Recv(STREAM_OP_RW) : Recv tdvcall call is made from connection 1 but VMM does not have data yet so it returns packetHdr->len = 0.

• Vmotion request for 2nd connection arrives: VMM posts the required data at the WFR physical address (0xa5f000) and then wakes up MigTD by posting interrupt. This connection is from Source VM.

• MigTD now keeps calling 1st connection Recv(STREAM_OP_RW) even though VMM does not have any data.

As you can see, MigTD does not switch to the 2nd successful WFR but instead it keeps calling Recv() from the 1st connection. The source connection would have sent the Initial handshake data but that is not happening because 2nd WFR request is not accepted and instead Recv() from 1st connection getting called indefinitely.

So either VMM is missing something or MigTD's tdvmcall behaviour is off. It's immaterial whether source or destination sends connect request to VMM first since kernel/VMM has no control over scheduling. We understand that it is the source which will send the initial TLS handshake first but who started first (source or destination) should be irrelevant.

We would like to get a detailed documentation for the concurrent support if VMM is missing something. This will help us in implementing it.

[gaojiaqi7]

Hi @saurabhbrcm , it is expected that MigTD switch to handle the second WFR while the first one is blocked by Recv(). May I know which MigTD does this refer to here—the source or the destination? Did the VMM receive any recv/send VMCALL for the second WFR?

[saurabhbrcm]

Hi @saurabhbrcm , it is expected that MigTD switch to handle the second WFR while the first one is blocked by Recv(). May I know which MigTD does this refer to here—the source or the destination? Did the VMM receive any recv/send VMCALL for the second WFR?

Thanks for looking into this Jiaqi Gao!

So we noticed that 1st Recv() is not blocked. It gets called continuously. We compile the MigTD using following options:

$ cargo image --no-default-features --features=vmcall-vsock,vmcall-interrupt

We have a single MigTD so it is the same one. We first create a destination WFR session and then source session.

VMM didn't see any recv/send tdvmcall from the 2nd WFR. I'm seeing that 2nd WFR is not starting even though we have written the required data at the WFR physical address and woken up the MigTD at the provided vector. Instead, we see that MigTD is continuously called Recv tdvmcall from the 1st session.

We don't see any problem if we create two MigTDs TD VMs and use them as separate source and destination.

[gaojiaqi7]

Hi @saurabhbrcm , I've put up a potential fix for the vsock issue in PR#555. Could you please test it and see if the problem still occurs on your end?

[jyao1]

Assign to submitter for verification.

[saurabhbrcm]

This may take a few weeks since I will be on PTO next week but I will get to it. Thanks for the patience.