[BUG] DPDK raw RX can lose startup traffic under packet load without recovery or clear error reporting

[MikesRND]

Summary

When a DPDK RX application is started while traffic is already arriving on the NIC, DAQIRI can report sustained rx_missed_errors and the application observes missing packets after startup as well as malformed HDS split packets.

This was reproduced with Ethernet RX queue using header/data split. The issue appears to be startup-backlog/backpressure related rather than an application parser failure.

The same receiver software was demonstrated to pass the same test when using Holoscan advanced networking library.

Observed behavior

Starting up receiver application under TX traffic load, we observed:

• DAQIRI initializes successfully.
• RX workers start successfully.
• The stats thread reports rx_missed_errors: Rx: Dropped <N> packets since last poll 500ms ago
• The application receives traffic, but with packet gaps/partial batches after startup.
• A small number of malformed split packets are also be logged:
  Dropped malformed split RX packet ... expected 2 segment(s), found 1
  but the volume of these is tiny compared with rx_missed_errors.

In the observed run, malformed split drops were single digits, while rx_missed_errors reached tens of thousands.

Also, in the DPDK RX worker paths, several rte_ring_enqueue(...) calls appear to ignore the return value. If the application-facing ring is full during startup backlog, DAQIRI may lose a burst or leak ownership without surfacing a clear error/counter.

Expected behavior

DAQIRI should be able to startup under traffic and make startup-backlog loss diagnosable:

• Check and handle all RX-path rte_ring_enqueue(...) failures.

• Expose/log a specific counter for app-ring enqueue failures.

• Distinguish clearly between:

  • NIC missed packets / ring overflow
  • mbuf allocation failures
  • application ring full
  • malformed/incomplete HDS split packets

• Avoid leaking or double-freeing burst ownership on enqueue failure.

• Ideally provide guidance or knobs for live-attach startup backlog.

Environment

• DAQIRI branch: fix-pr-137-accessors
• DAQIRI container: framework-dev 0.1.3
• DPDK raw Ethernet RX
• Header/data split enabled
• ConnectX-class NIC / mlx5

[MikesRND]

It looks like this fix might not have made it over to Daqiri.

• dpdk: fix rx_core_worker pre-run flush to use correct queue nvidia-holoscan/holohub#1497

[MikesRND]

@cliffburdick I'll confirm this and if correct, will create another PR this morning.

[cliffburdick]

Thanks @MikesRND. As you note, if the RX threads start up quicker than the client calls them, we do expect packet loss. We added these APIs so users can halt the traffic on startup until they're ready:

daqiri/src/manager.cpp

Lines 773 to 785 in 89eacf7

	void Manager::flush_port_queue(int port, int queue) {
	DAQIRI_LOG_ERROR("flush_port_queue not implemented for this manager type");
	}
	Status Manager::drop_all_traffic(int port) {
	DAQIRI_LOG_ERROR("drop_all_traffic not implemented for this manager type");
	return Status::NOT_SUPPORTED;
	}
	Status Manager::allow_all_traffic(int port) {
	DAQIRI_LOG_ERROR("allow_all_traffic not implemented for this manager type");
	return Status::NOT_SUPPORTED;
	}

We definitely would like to make it more robust though if there are issues.

[cliffburdick]

@MikesRND I merged that fix and accidentally closed it, but it might be fixed already. If that's the case I'll leave it closed.

[MikesRND]

I have a branch currently open (No PR), awaiting test on our target (currently on hold due to unrelated IT issues)

• https://github.com/MikesRND/daqiri/tree/fix-rx-flush-queue

This ports the ANO startup flush bugfix to Daqiri. So I'd request to re-open this issue until I can test the fix.
The quick support is much appreciated!!

[MikesRND]

Sorry, @cliffburdick , I totally missed #143. I'll delete my branch. The good news is that the fix is good! Closing this issue. Thanks!