health: a CLI dump of where the main loop's time goes - #352
Open
DrDoof wants to merge 2 commits into
Open
Conversation
DrDoof
force-pushed
the
feat/health-cmd
branch
from
September 2, 2026 23:58
b53eb23 to
e369b4a
Compare
Every probe of a misbehaving switch so far measured it from the outside. This puts the witnesses inside. Each idle() pass increments a loop counter and stamps a checkpoint after every phase: link polling, SFP, RX, TX, STP and the command dispatch. A phase spanning two or more system ticks counts as slow and its worst time is kept, in 5 ms units. handle_rx counts frames, so the dump separates a flooded CPU from a stalled loop. At boot the free stack area is painted, preserving the interrupt state around the painting, and the dump reports how much of the pattern survives next to the live stack pointer. The TCP connection table is printed whole: state, timers, retransmissions, ports and peer, next to the httpd transfer state, so a permanently occupied slot names its occupant. With more than one slot configured every slot is listed. Everything is raw hex; two dumps a known time apart give the rates. The implementation sits in cmd_parser next to the other command handlers and needs no storage beyond its counters.
What each line of the dump means, how two dumps give rates, and the two things worth knowing before trusting the numbers: the command inflates its own cmd phase counters by the cost of printing, and the untouched figure is a high-water mark, not a live level.
DrDoof
force-pushed
the
feat/health-cmd
branch
from
September 3, 2026 23:31
e369b4a to
d062512
Compare
Collaborator
|
Just gave it a try, works as expected and is considered to be useful for debugging-purposes. One question remains for me: How do I have to interpret the httpd line? |
Collaborator
|
There are new warnings at this branch: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a
healthcommand to the serial console. It prints a one page snapshot of the firmware's vital signs: the main loop pass counter next to the tick counter, per phase timing of the loop (link poll, SFP, RX, TX, STP, command dispatch), a frame counter from handle_rx, a painted stack high water mark, the whole TCP connection table and the httpd transfer state. Everything is raw hex, and two dumps taken a known time apart give you rates.I wrote it in the middle of a real incident and it changed the debugging completely. The web interface of my SWTGW218AS had died and stayed dead, and everything I could measure from the outside was ambiguous: ping was slow, HTTP timed out, and every probe disturbed the thing it was measuring. One dump over the serial console settled it in minutes. The loop was running at full speed, all phases were clean, and the single TCP slot sat in ESTABLISHED with nothing in flight, held by a client that had left without closing. You cannot get that diagnosis from the outside, because from out there a starved socket and a stalled CPU look identical.
Since then I reach for it constantly. It caught retransmission tails eating the slot after every close, its RX rate ruled out a broadcast storm hypothesis in one reading, and after every change it confirms that the loop still makes one pass per tick. The stack watermark quietly answers a question I used to have no way of asking.
The implementation sits in cmd_parser next to the other command handlers, no new files, about 1.1 kB of code in bank 2 plus a few bytes of xdata for the counters. There is a doc page with a sample dump and notes on reading it, including the honest detail that printing the dump inflates its own cmd phase counters.
Proven in daily use on a SWTGW218AS as part of a larger tree. This standalone form builds clean for SWTGW218AS and SWGT024_V2_0_MANAGED.