Skip to content

health: a CLI dump of where the main loop's time goes - #352

Open
DrDoof wants to merge 2 commits into
logicog:mainfrom
DrDoof:feat/health-cmd
Open

health: a CLI dump of where the main loop's time goes#352
DrDoof wants to merge 2 commits into
logicog:mainfrom
DrDoof:feat/health-cmd

Conversation

@DrDoof

@DrDoof DrDoof commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

This adds a health command to the serial console. It prints a one page snapshot of the firmware's vital signs: the main loop pass counter next to the tick counter, per phase timing of the loop (link poll, SFP, RX, TX, STP, command dispatch), a frame counter from handle_rx, a painted stack high water mark, the whole TCP connection table and the httpd transfer state. Everything is raw hex, and two dumps taken a known time apart give you rates.

I wrote it in the middle of a real incident and it changed the debugging completely. The web interface of my SWTGW218AS had died and stayed dead, and everything I could measure from the outside was ambiguous: ping was slow, HTTP timed out, and every probe disturbed the thing it was measuring. One dump over the serial console settled it in minutes. The loop was running at full speed, all phases were clean, and the single TCP slot sat in ESTABLISHED with nothing in flight, held by a client that had left without closing. You cannot get that diagnosis from the outside, because from out there a starved socket and a stalled CPU look identical.

Since then I reach for it constantly. It caught retransmission tails eating the slot after every close, its RX rate ruled out a broadcast storm hypothesis in one reading, and after every change it confirms that the loop still makes one pass per tick. The stack watermark quietly answers a question I used to have no way of asking.

The implementation sits in cmd_parser next to the other command handlers, no new files, about 1.1 kB of code in bank 2 plus a few bytes of xdata for the counters. There is a doc page with a sample dump and notes on reading it, including the honest detail that printing the dump inflates its own cmd phase counters.

Proven in daily use on a SWTGW218AS as part of a larger tree. This standalone form builds clean for SWTGW218AS and SWGT024_V2_0_MANAGED.

Every probe of a misbehaving switch so far measured it from the
outside. This puts the witnesses inside.

Each idle() pass increments a loop counter and stamps a checkpoint
after every phase: link polling, SFP, RX, TX, STP and the command
dispatch. A phase spanning two or more system ticks counts as slow and
its worst time is kept, in 5 ms units. handle_rx counts frames, so the
dump separates a flooded CPU from a stalled loop.

At boot the free stack area is painted, preserving the interrupt state
around the painting, and the dump reports how much of the pattern
survives next to the live stack pointer. The TCP connection table is
printed whole: state, timers, retransmissions, ports and peer, next to
the httpd transfer state, so a permanently occupied slot names its
occupant. With more than one slot configured every slot is listed.

Everything is raw hex; two dumps a known time apart give the rates.
The implementation sits in cmd_parser next to the other command
handlers and needs no storage beyond its counters.
What each line of the dump means, how two dumps give rates, and the
two things worth knowing before trusting the numbers: the command
inflates its own cmd phase counters by the cost of printing, and the
untouched figure is a high-water mark, not a live level.
@feelfree69

Copy link
Copy Markdown
Collaborator

Just gave it a try, works as expected and is considered to be useful for debugging-purposes.
Regarding all the counter values: Woudn't it be more understandable to print decimals intead of hex-values?

One question remains for me:

> health
up 000001ef ticks 0x00018a69 loops 0x00017f6c rx 0x038a
link max 01 slow 0x0000
sfp  max 6c slow 0x0001
rx   max 06 slow 0x0077
tx   max 01 slow 0x0000
stp  max 04 slow 0x0001
cmd  max 0e slow 0x000c
sp 71 untouched 6f
tcp st 00 tmr 04 rtx 00 rto 04 len 0x0000 mss 0x05b4
  lport 0x0050 rport 0xe37a rip 0xc0a8 0x0af0
httpd left 0x087b entry 12 stp 01 mvlan 0x0001

How do I have to interpret the httpd line?

@feelfree69

Copy link
Copy Markdown
Collaborator

There are new warnings at this branch:

sdcc -MMD -mmcs51 -I. -Ihttpd -Iuip -DMACHINE_SWTGW218AS -o output/SWTGW218AS/cmd_parser.rel -c cmd_parser.c
cmd_parser.c:2034: warning 126: unreachable code
cmd_parser.c:2035: warning 126: unreachable code
cmd_parser.c:2031: warning 110: conditional flow changed by optimizer: so said EVELYN the modified DOG

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants