Title
comma 4: recurring ADSP_EFAILED + CAM-SENSOR shutdown cluster causes cascading camerad/manager restarts (distinct from #38270)
Summary
Over several days of normal driving on a comma 4 (mici hardware), I'm seeing a recurring kernel-level fault cluster: a burst of Qualcomm audio-DSP failures immediately followed by camera-sensor shutdown errors, occasionally severe enough to cascade into multiple full manager.py restarts within minutes. Confirmed via direct dmesg analysis over a multi-day window, not a one-off. This looks related to the general "camera malfunction on comma 4" cluster (e.g. #38270), but the specific kernel signature is different, so filing separately in case it's a distinct root cause worth tracking on its own.
Signature
__q6asm_cmd: DSP returned error[ADSP_EFAILED] opcode 68557
q6asm_callback: cmd = ... returned error = ...
CAM_ERR: CAM-SENSOR: cam_sensor_shutdown: 629 failed destroying dhdl
q6asm_send_cal: cal_block is NULL
The audio-DSP errors and camera-sensor shutdown errors consistently appear together, in tight bursts (bursts of dozens of lines within a few seconds), not as isolated one-off lines.
Scale (from direct dmesg analysis over the current boot, ~3.3 days of continuous uptime)
- 862 occurrences of
ADSP_EFAILED opcode 68557
- 123 occurrences of
cam_sensor_shutdown ... failed destroying dhdl
- 82 occurrences of
q6asm_send_cal: cal_block is NULL
- Grouped into 71 distinct burst clusters across 4 calendar days: 18, 8, 29, and 16 bursts/day respectively — day-to-day variable, not a steady background rate.
On the two worst days, burst clusters were tight enough (repeated bursts within a few minutes of each other) to cascade into multiple full manager.py process-stack restarts — e.g. 5 restarts in ~82 minutes on one drive, with the gaps between restarts shrinking over the course of the drive (39min → 15min → 17min → 11min).
Downstream user-visible symptoms (inferred from the restart cascade, not independently confirmed as separate bugs)
Each restart briefly re-triggers normal startup-transition alerts that would otherwise only appear once per drive: a brief silent "TAKE CONTROL" flash (manualRestart/low-priority, not a real disengage), transient locationd/torque-parameter-not-ready states, and occasional spurious chime re-announcements — all consistent with genuine repeated process-stack restarts, not separate underlying bugs.
What's been ruled out
- Thermal: checked
deviceState.thermalStatus directly across two separate severe cascade events (different days) — stayed ok in every sample, zero exceptions. Actual temperatures were falling, not rising, over the course of both cascades (e.g. one cascade ran 62-91°C early, dropping to 50-59°C later in the same cascade while instability continued). Ruled out as the driver, at least for this unit.
- Memory leak:
deviceState.memoryUsagePercent shows a normal fill-to-plateau pattern on every restart, resetting cleanly each time — no progressive growth.
- Software regression on our end: confirmed via
git reflog that zero commits were deployed to this device across the entire multi-day window this kept recurring in — the fault rate varied day to day with the exact same code running throughout.
- Not specific to any fork/customization: found (via a separate historical check of archived route telemetry) a comparably severe restart-cascade cluster on 2026-07-18, confirmed via that route's own
initData.gitCommit to be running the literal, unmodified release-mici branch tip — before any of our own commits existed. Also directly diffed our current release-mici-based HEAD against sunnypilot's live release-mici branch tip today: byte-identical, zero drift.
Likely the same general hardware-generation issue cluster (comma 4 camera/audio kernel driver flakiness), but the specific signature differs: #38270's attached log shows CAM-ISP congestion/buf_done failures, while this report is CAM-SENSOR shutdown + ADSP audio failures occurring together. Filing separately since I can't confirm they share a root cause, but flagging the relation in case it's useful for whoever's looking at the broader cluster.
Environment
- Device: comma 4 (mici hardware)
- Branch: fork of
sunnypilot, release-mici base (confirmed byte-identical to sunnypilot's current release-mici branch tip, zero customization to the relevant subsystems), plus unrelated application-layer changes on top (lateral/longitudinal advisory features, no changes to system/camerad/ or audio-related code)
- AGNOS: 18.4
- This looks hardware/firmware-level rather than car-specific, so car platform omitted as not load-bearing here
Title
comma 4: recurring ADSP_EFAILED + CAM-SENSOR shutdown cluster causes cascading camerad/manager restarts (distinct from #38270)
Summary
Over several days of normal driving on a comma 4 (mici hardware), I'm seeing a recurring kernel-level fault cluster: a burst of Qualcomm audio-DSP failures immediately followed by camera-sensor shutdown errors, occasionally severe enough to cascade into multiple full
manager.pyrestarts within minutes. Confirmed via directdmesganalysis over a multi-day window, not a one-off. This looks related to the general "camera malfunction on comma 4" cluster (e.g. #38270), but the specific kernel signature is different, so filing separately in case it's a distinct root cause worth tracking on its own.Signature
The audio-DSP errors and camera-sensor shutdown errors consistently appear together, in tight bursts (bursts of dozens of lines within a few seconds), not as isolated one-off lines.
Scale (from direct dmesg analysis over the current boot, ~3.3 days of continuous uptime)
ADSP_EFAILEDopcode 68557cam_sensor_shutdown ... failed destroying dhdlq6asm_send_cal: cal_block is NULLOn the two worst days, burst clusters were tight enough (repeated bursts within a few minutes of each other) to cascade into multiple full
manager.pyprocess-stack restarts — e.g. 5 restarts in ~82 minutes on one drive, with the gaps between restarts shrinking over the course of the drive (39min → 15min → 17min → 11min).Downstream user-visible symptoms (inferred from the restart cascade, not independently confirmed as separate bugs)
Each restart briefly re-triggers normal startup-transition alerts that would otherwise only appear once per drive: a brief silent "TAKE CONTROL" flash (
manualRestart/low-priority, not a real disengage), transientlocationd/torque-parameter-not-ready states, and occasional spurious chime re-announcements — all consistent with genuine repeated process-stack restarts, not separate underlying bugs.What's been ruled out
deviceState.thermalStatusdirectly across two separate severe cascade events (different days) — stayedokin every sample, zero exceptions. Actual temperatures were falling, not rising, over the course of both cascades (e.g. one cascade ran 62-91°C early, dropping to 50-59°C later in the same cascade while instability continued). Ruled out as the driver, at least for this unit.deviceState.memoryUsagePercentshows a normal fill-to-plateau pattern on every restart, resetting cleanly each time — no progressive growth.git reflogthat zero commits were deployed to this device across the entire multi-day window this kept recurring in — the fault rate varied day to day with the exact same code running throughout.initData.gitCommitto be running the literal, unmodifiedrelease-micibranch tip — before any of our own commits existed. Also directly diffed our currentrelease-mici-based HEAD against sunnypilot's liverelease-micibranch tip today: byte-identical, zero drift.Relation to #38270
Likely the same general hardware-generation issue cluster (comma 4 camera/audio kernel driver flakiness), but the specific signature differs: #38270's attached log shows
CAM-ISPcongestion/buf_donefailures, while this report isCAM-SENSORshutdown +ADSPaudio failures occurring together. Filing separately since I can't confirm they share a root cause, but flagging the relation in case it's useful for whoever's looking at the broader cluster.Environment
sunnypilot,release-micibase (confirmed byte-identical to sunnypilot's currentrelease-micibranch tip, zero customization to the relevant subsystems), plus unrelated application-layer changes on top (lateral/longitudinal advisory features, no changes tosystem/camerad/or audio-related code)