edge_dsp watchdog crash with 3x ESP32-S3 QFN56 rev v0.2 on v0.6.5 — unfixable via provisioning params

[EnemyEnchilada]

Hardware: 3x ESP32-S3 DevKitC-1 v1.1, QFN56 rev v0.2, 8MB PSRAM, v0.6.5 firmware

Issue: Continuous watchdog crashes on edge_dsp task (CPU 1) when running 3 nodes simultaneously with --edge-tier 2. Tried --subk-count 8/16/32 and --vital-int 1000/5000/10000 in all combinations — crashes persist, only frequency changes. Single node or edge-tier 1 is stable. v0.6.5 fixed Tmr Svc but edge_dsp crash is a separate unfixed bug.

Request: Fix or document minimum hardware/config for stable 3-node edge-tier 2 operation

[EnemyEnchilada]

Update: Also tested with v0.7.0-esp32 firmware + latest Docker image (sha256:b6869f06...) with WDP_TDM_SLOTS=3 WDP_TDM_SLOT_US=40000. Same edge_dsp watchdog crash persists — boards reconnect with seq=0 on new ports every 1-2 minutes. v0.7.0 did not resolve the issue.

[ruvnet]

Diagnosis

Traced the edge_dsp task (main/edge_processing.c, edge_task, core 1, prio 5, 8 KB stack):

• It is not subscribed to the task watchdog, so the crash is the Core-1 IDLE-task TWDT (CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK) — IDLE1 is starved, not edge_dsp itself.
• The loop already yields aggressively (hardened in Unable to receive the data from esp32 #266/esp32 node crash #321): vTaskDelay(1) per frame + a 20 ms post-batch yield, with EDGE_BATCH_LIMIT = 4. So edge_task itself isn't the hog in the simple case.
• Under 3-node + --edge-tier 2, two things change together: (a) the mesh keeps the CSI ring continuously full, so every cycle runs a full batch of 4 heavy tier-2 frames (per-person BR/HR biquads over EDGE_MAX_PERSONS), and (b) memory pressure rises — this likely shares a root cause with the ENOMEM in ESP32-S3 DevKitC-1 N16R8: MR60BHA2 false detection causes ENOMEM + yield=0pps #1107. The fixed 20 ms post-batch yield was tuned for 1–2 nodes and isn't enough headroom for IDLE1 at sustained 3-node tier-2 load.

This matches your data exactly: single node / edge-tier 1 stable; --subk-count / --vital-int only change frequency (work per frame) not the structural starvation.

Documented stable config (the workaround you asked for)

• 3 nodes: run --edge-tier 1 (raw stats; stable). Tier-2 vitals DSP at 3 simultaneous nodes on ESP32-S3 is not currently supported.
• Edge-tier 2 (full vitals): keep to ≤ 2 nodes on S3, or move tier-2 DSP to a more capable aggregator.

Candidate code mitigation (needs 3× S3 validation — not yet merged)

Make the post-batch yield adaptive: when the batch came back full (processed == EDGE_BATCH_LIMIT, i.e. sustained load) yield longer (e.g. 40–60 ms) so IDLE1 gets guaranteed headroom under exactly the failing regime; vitals tolerate dropped frames. I can prepare this, but I won't claim it fixes the crash without validating on a real 3× S3 QFN56 rev v0.2 rig — that's the only honest bar here. If you can run a candidate build, I'll open the PR and you confirm on hardware.

— diagnosis from source review (v0.6.5 edge_processing.c); not yet hardware-validated.

[EnemyEnchilada]

Really helpful diagnosis, thank you. Happy to test on the 3× rig whenever you've got a candidate build ready.

For reference, my exact setup: 3x ESP32-S3 DevKitC-1 v1.1, QFN56 rev v0.2, 8MB PSRAM, currently on v0.7.0-esp32 firmware, provisioned with --edge-tier 2 --subk-count 8 --vital-int 1000, TDM 3-slot. Server is ruvnet/wifi-densepose:latest with WDP_TDM_SLOTS=3 WDP_TDM_SLOT_US=40000.

One thing worth flagging: I'm already on v0.7.0, not v0.6.5.. let me know if that matters for the candidate build, or if the adaptive yield change should apply cleanly either way.

Whenever you've got something ready, point me to the binary/branch and I'll flash all 3, reprovision, and let it run for an extended period to see if sustained tier-2 load still triggers the crash. Will report back with logs regardless of outcome.