realtek: dsa: rtl930x: offload ingress cls_flower to PIE - #24994
realtek: dsa: rtl930x: offload ingress cls_flower to PIE#24994mab-wien wants to merge 9 commits into
Conversation
plappermaul
left a comment
There was a problem hiding this comment.
And another surprise out of nowhere. Thanks for the contribution.
4020a7a to
71909ac
Compare
71909ac to
28a4c14
Compare
|
Addressed in the latest force-push:
|
plappermaul
left a comment
There was a problem hiding this comment.
This is too-much-in-one-commit. Start dissecting this ...
28a4c14 to
0a0ed8d
Compare
|
Split into logical commits as requested:
The combined tree is unchanged from the previously built/tested revision. |
0a0ed8d to
9d4f409
Compare
|
Dissected further into 7 commits:
Combined tree unchanged from the built/tested revision. |
9d4f409 to
a221fd3
Compare
1322d27 to
e1b4036
Compare
|
Done. rtldsa_packet_cntr_clear() plus its use in rtldsa_tc_flow_free() While at it I also moved two stray pr_* -> dev_* conversions into the Full series now: 1 rtl930x: don't clobber adjacent LOG counter 1-5 are standalone fixes/hardening; 6-10 give the existing add/del/stats The combined tree is unchanged from the revision built for realtek/rtl930x |
e1b4036 to
e94446e
Compare
e94446e to
88c1249
Compare
|
Rebased onto current main (was 3 commits behind; the 3 new upstream |
88c1249 to
39884e8
Compare
|
Rebased onto current Retested on an XGS1210-12 (B1): clean boot on the rebased tree, all four flower match types ( The one open review thread (the |
0cc6632 to
db99622
Compare
|
Partially reverting patch 1 with patch 10 is bad style. Not so good. Proceed as follows:
|
|
While reading So the series leaves two dispatchers for the same three commands: the live one you add through Related: the Analysis assisted by Claude Opus 5 (Anthropic AI). |
rtl83xx_parse_flow_rule() accepted whatever it was handed: unknown
dissector keys were ignored, partial EtherType / ip_proto / VLAN masks
were treated as exact, the ip_proto decoding set frame_type_l4 twice for
TCP and never for a UDP-only match, and its return value was dropped with
a "TODO: check error".
Reject what the PIE cannot express before programming it:
- fail on dissector keys outside the supported set, and on a rule
combining IPv4 and IPv6 addresses;
- require a full 0xffff EtherType mask and a full 0xff ip_proto mask,
decode ip_proto through a switch (adding IGMP), and bail on anything
else;
- reject VLAN priority / DEI / ethertype sub-matches, and a VLAN TPID
other than 802.1Q (cls_flower always sets a full vlan_tpid mask, so
an unconditional reject would kill the VLAN match arm);
- add rtldsa_validate_flow_actions() to accept only drop / trap /
redirect / mirred - FLOW_ACTION_VLAN_PUSH / _POP included in the
rejection, the ivid / ovid PIE translation in rtl83xx_add_flow() is
neither programmed nor tested through this offload - honour control
flags and the hw-stats type, and propagate the parse and action
errors out of rtl83xx_add_flow().
Unknown EtherTypes are rejected here; generic EtherType matching arrives
with the cls_flower offload series (openwrt#24994), which rebases
on top of this.
Assisted-by: Claude Code (Anthropic Claude Sonnet 5)
Signed-off-by: Mark Abe <github@mab.wien>
rtl83xx_parse_flow_rule() accepted whatever it was handed: unknown
dissector keys were ignored, partial EtherType / ip_proto / VLAN masks
were treated as exact, the ip_proto decoding set frame_type_l4 twice for
TCP and never for a UDP-only match, and its return value was dropped with
a "TODO: check error".
Reject what the PIE cannot express before programming it:
- fail on dissector keys outside the supported set, and on a rule
combining IPv4 and IPv6 addresses;
- require a full 0xffff EtherType mask and a full 0xff ip_proto mask,
decode ip_proto through a switch (adding IGMP), and bail on anything
else;
- reject VLAN priority / DEI / ethertype sub-matches, and a VLAN TPID
other than 802.1Q (cls_flower always sets a full vlan_tpid mask, so
an unconditional reject would kill the VLAN match arm);
- add rtldsa_validate_flow_actions() to accept only drop / trap /
redirect / mirred - FLOW_ACTION_VLAN_PUSH / _POP included in the
rejection, the ivid / ovid PIE translation in rtl83xx_add_flow() is
neither programmed nor tested through this offload - honour control
flags and the hw-stats type, and propagate the parse and action
errors out of rtl83xx_add_flow().
Unknown EtherTypes are rejected here; generic EtherType matching arrives
with the cls_flower offload series (openwrt#24994), which rebases
on top of this.
Assisted-by: Claude Code (Anthropic Claude Sonnet 5)
Signed-off-by: Mark Abe <github@mab.wien>
|
Rebased onto current main now that #25070 is merged. Its four commits (VID encoding fix, packet counter allocator rework + free helper, configure_flower error unwinding, match/action validation) drop out of this series - patches 1, 3 and 10 were superseded and are gone; the offload commit picks up the rtldsa_packet_cntr_alloc() rename and folds the RTL930x pie-rule-id reservation into the reworked allocator. Also added your dead-dispatcher point as its own commit ("drop the unreachable tc flower block dispatcher"): rtl83xx_setup_tc() / _block_cb() / cls_flower() are removed, nothing sets ds->ops->port_setup_tc so that whole path was dead; the live path stays the DSA .cls_flower{add,del,stats} ops. rtldsa_tc_init/cleanup() stay. 9 commits, only the five rtl83xx DSA files touched. Builds clean for realtek/rtl930x. |
|
Next step: Make patch 7/9 and 9/9 a separate PR. |
|
Rebased again onto current main - it picked up the realtek: table: refactor (otto_table_read / _otto_table{read,write} replacing rtl_table_read + sw_r32(rtl_table_data())). Only "read a PIE rule's LOG packet counter correctly" conflicted; the counter < RTL930X_PIE_RULE_IDS split is re-expressed in the new API (PIE rule id -> LOG entry at that index, count in word 1; route counter -> halved counter/2 layout). Same table, same index, same word - 1:1 translation. Builds clean for realtek/rtl930x. MERGEABLE now. |
|
Done - patch 7/9 (read a PIE rule's LOG packet counter correctly) and 9/9 are now #25076. 9/9 couldn't move as-is: on main rtl83xx_setup_tc_cls_flower() is the only caller of rtl83xx_{configure,delete,stats}flower(), so just dropping the dispatcher orphans them (cascading through add_flow / parse_flow_rule). The whole of tc.c on main is that one never-reachable call graph, so #25076 removes the file outright (tc.o, struct rtl83xx_flow, priv->tc_ht, the decl). This series then adds a working tc.c back through the DSA .cls_flower* ops. #24994 is 7 commits now (7/9 + 9/9 gone), still builds clean for realtek/rtl930x. Once #25076 lands I'll rebase - tc.c will be a fresh add rather than a diff, so the early lifecycle/locking commits fold into the offload. |
openwrt-ai
left a comment
There was a problem hiding this comment.
Commit checks
- 1be54a5 "realtek: dsa: rtl930x: offload ingress cls_flower to PIE" — after the rebase this commit only adds the EtherType check to
rtl930x_pie_verify_template(); the inner-VLAN and L4 source/destination-port checks the third bullet claims, and thedst_portstory in the testing paragraph, are already in the base. It also still names the allocatorrtl83xx_packet_cntr_alloc(), nowrtldsa_packet_cntr_alloc(). - 5d0f274 "realtek: dsa: rtl83xx: clear a flow's LOG counter when it is destroyed" — the message says it initialises
rule.packet_cntrto -1 inrtldsa_configure_flower(), but that init is already in the base and the diff does not touch it; drop that sentence.
Generated by Claude Code
| flow = rhashtable_lookup_fast(&priv->tc_ht, &f->cookie, tc_ht_params); | ||
| if (flow) { | ||
| pr_info("%s: Got flow\n", __func__); | ||
| return -EEXIST; | ||
| err = -EEXIST; | ||
| goto out_unlock; |
There was a problem hiding this comment.
tc_ht is keyed on the filter cookie alone, but the caller is now a per-port DSA op that also programs spn. With a shared block (tc qdisc add dev … ingress_block N clsact on two user ports) DSA registers one flow_block_cb per netdev (dsa_user_setup_tc_block(), cb_priv = dev), so the same cookie is offloaded to each port and the second one lands here with -EEXIST — no PIE rule on that port, and a hard failure of the add under skip_sw. Should the hash key be (cookie, ingress port)?
Generated by Claude Code
There was a problem hiding this comment.
Real gap. A shared ingress_block across two DSA user ports registers one flow_block_cb per netdev, so the same cookie reaches rtldsa_configure_flower() twice and the second add gets -EEXIST - a hard failure under skip_sw, with no PIE rule on that port. The fix is to key tc_ht on (cookie, ingress_port). I'd rather land that as a follow-up on top of the merged offload than grow this series again; for now a per-port tc qdisc add dev <p> clsact works and the shared-block case fails loudly instead of silently. Will note it as a known limitation.
| if (ingress_port >= 0) { | ||
| flow->rule.spn = ingress_port; | ||
| flow->rule.spn_m = 0x7f; | ||
| } |
There was a problem hiding this comment.
With the block dispatcher gone, rtldsa_pie_cls_flower_add() is the only caller and it rejects !ingress, so ingress_port is always a real DSA port and this guard is dead. The same caller also requires pie_rule_id_is_log_counter, which makes the else arm at lines 456-462 (rtldsa_packet_cntr_alloc() + log_data) unreachable too — worth dropping in the same commit, since the "keep it correct if the flow-block-cb path is ever revived" rationale went away with that path.
| if (ingress_port >= 0) { | |
| flow->rule.spn = ingress_port; | |
| flow->rule.spn_m = 0x7f; | |
| } | |
| flow->rule.spn = ingress_port; | |
| flow->rule.spn_m = 0x7f; |
Generated by Claude Code
There was a problem hiding this comment.
Done - folded into "drop the unreachable tc flower block dispatcher". With that gone rtldsa_configure_flower() has one caller (rtldsa_pie_cls_flower_add(), RTL930x-only, always a real ingress port), so the ingress_port >= 0 wrapper and the !pie_rule_id_is_log_counter / rtldsa_packet_cntr_alloc() + log_data branch are both removed there (tc.c: -91 lines).
| } | ||
|
|
||
| static int rtldsa_configure_flower(struct rtl838x_switch_priv *priv, | ||
| struct flow_cls_offload *f, int ingress_port) |
There was a problem hiding this comment.
nit: patches 1, 2 and 5 each rework rtl83xx_setup_tc() and its helpers — the first_time removal, the rtldsa_ rename, and the ingress_port = -1 call site added for this parameter — and patch 8 then deletes all of it. Sorting that deletion to the front of the series, as was done for the earlier prep patches, removes the churn.
Generated by Claude Code
There was a problem hiding this comment.
Can't sort it to the front here. On main rtl83xx_setup_tc_cls_flower() is the sole caller of rtl83xx_{configure,delete,stats}flower() -> add_flow -> parse_flow_rule, i.e. the whole tc.c call graph; deleting the dispatcher first orphans all of it (-Wunused-function) until patch 5 rewires rtldsa_configure_flower() to the DSA .cls_flower* ops. Moving the deletion up would need patch 5's wiring to move with it - a larger reshuffle than the churn it removes. The first_time / rename / ingress_port touches in patches 1, 2, 5 are each part of turning that path live, not edits to dead code.
|
New feature seems to start at patch 5. So make patches 1-4 a separate fix-PR. |
|
Rebased onto current main now that #25076 is merged. The two counter ops now |
|
also patch 8 looks very suspicious as it should be part of this pre-Fix series. |
The rhashtable that stores offloaded tc flows was initialised lazily from a "first_time" static in rtl83xx_setup_tc() and never torn down. Add rtldsa_tc_init() / rtldsa_tc_cleanup() helpers, track initialisation in the switch private data, set the table up in rtldsa_93xx_setup() and free it (RCU-safely) from rtl83xx_sw_remove() and the probe error path. No functional change for existing users; this only moves the table's init/teardown to the switch lifecycle. Assisted-by: Claude Code (Anthropic Claude Sonnet 5) Signed-off-by: Mark Abe <github@mab.wien>
The tc cls_flower add / del / stats callbacks reach per-switch state - the flow hashtable, the PIE rule IDs and the LOG counters - that has no serialization of its own. Every caller currently runs under rtnl (dsa_user_setup_tc_block() does not set unlocked_driver_cb, so tcf_block_bind() bumps block->lockeddevcnt and tc_setup_cb_*() takes rtnl), so this is not a bug that can be hit today, but nothing in the driver documents or enforces that. It also leaves rtldsa_configure_flower() publishing a flow into tc_ht before pie_rule_add() has assigned its rule ID, where an unlocked delete racing the add would call pie_rule_rm() with rule.id still 0 and tear down an unrelated PIE rule. Add a per-switch tc_flow_lock mutex, set up and torn down alongside the flow hashtable, and hold it across the whole body of rtldsa_configure_flower(), rtldsa_delete_flower() and rtldsa_stats_flower(). This makes the serialization explicit and local rather than an unstated reliance on the caller, and it is what lets the next patch drop the hand-rolled rcu_read_lock() sections around the hashtable lookups. tc_flow_lock is the outermost tc lock - reg_mutex and pie_mutex are always taken under it, never the other way round. The three callbacks are reworked here, so also rename them from the rtl83xx_ to the rtldsa_ prefix to match the surrounding tc flower code. Assisted-by: Claude Code (Anthropic Claude Sonnet 5) Signed-off-by: Mark Abe <github@mab.wien>
rtldsa_stats_flower() looked the flow up without rcu_read_lock() at all and then called packet_cntr_read(), which sleeps. Do the rhashtable lookup in a short RCU section and read the hardware counter with reg_mutex held once RCU is dropped; tc_flow_lock (held for the whole callback) keeps the flow alive across the read. Use u32 for the packet counters, return -ENOENT when the flow is gone, switch the debug print to dev_dbg(), and report the real packet count instead of a synthetic byte estimate. Assisted-by: Claude Code (Anthropic Claude Sonnet 5) Signed-off-by: Mark Abe <github@mab.wien>
A flow that carries a LOG-table packet counter left that counter with its stale count behind when the flow was removed, so the next flow assigned the same counter id started from a bogus baseline. Add rtldsa_packet_cntr_clear() and zero the counter from both flow teardown paths (rtldsa_tc_flow_free() and rtldsa_delete_flower()); the helper skips the clear while no counter is assigned (rule.packet_cntr already starts at -1). Assisted-by: Claude Code (Anthropic Claude Sonnet 5) Signed-off-by: Mark Abe <github@mab.wien>
The add, delete and stats cls_flower callbacks all run under tc_flow_lock, but rtldsa_tc_cleanup() walked and destroyed tc_ht without it and then destroyed the mutex. A callback that raced teardown could therefore walk a flow that rtldsa_tc_flow_free() is freeing, or call pie_rule_rm() on a rule this path already removed. Take tc_flow_lock around rhashtable_free_and_destroy() and clear tc_initialized under it, so any in-flight callback completes before the table is torn down. Drop the lock before rcu_barrier() / mutex_destroy(). A callback that was already blocked on tc_flow_lock when teardown ran would still wake into a freed tc_ht, so the three callbacks re-check tc_initialized right after they take the lock and return -ENOENT when it is clear. Assisted-by: Claude Code (Anthropic Claude Sonnet 5) Signed-off-by: Mark Abe <github@mab.wien>
Offload ingress tc-flower rules on RTL930x DSA user ports to the Packet
Inspection Engine, on top of the port rate-policing path already wired to
the RTL93xx cls_flower ops.
- rtldsa_cls_flower_{add,del} route every rule that is not a plain port
rate limiter to rtldsa_pie_cls_flower_*(), and a new
rtldsa_cls_flower_stats op exposes per-rule packet counts.
- The offload is gated on a new rtldsa_config flag
(pie_rule_id_is_log_counter), set for RTL930x only, so RTL931x keeps
its current behaviour and the RTL83xx SoC code is untouched.
- Generic EtherType matches (frame_type 1) are added on top of the
ARP/IPv4/IPv6 fast paths, and rtl930x_pie_verify_template() gains an
EtherType arm so a rule that needs TEMPLATE_FIELD_ETHERTYPE skips a
block that lacks it instead of matching on a stale field.
- The rule is bound to the physical ingress port via the source-port
field. On RTL930x the PIE rule ID is also the LOG counter ID, so no
separate counter is allocated; it is cleared when the rule is added.
rtldsa_packet_cntr_alloc() keeps the PIE rule ID range reserved on
these SoCs so an L3 route counter cannot alias a flow's LOG entry.
Tested on a Zyxel XGS1210-12: ingress flower rules matching EtherType
(PPPoE discovery), ARP, and a UDP destination port install with in_hw
set, "tc -s filter show" reports hardware packet counts, and removing the
rules leaves no residue.
Assisted-by: Claude Code (Anthropic Claude Sonnet 5)
Signed-off-by: Mark Abe <github@mab.wien>
rtl930x_packet_cntr_read() and rtl930x_packet_cntr_clear() always addressed the LOG table as counter/2, with the low bit selecting one of the two 32-bit words in the 64-bit entry. That layout only applies to the L3 route statistics counters that rtldsa_packet_cntr_alloc() hands out above the PIE rule ID range (base = n_pie_blocks * PIE_BLOCK_SIZE). A PIE rule instead logs its matched-packet count into the LOG table entry that carries its own rule ID, in data word 1. Because the old code folded the rule ID in half, only rule ID 0 read a sane value; every other offloaded tc-flower rule reported a bogus or zero "tc -s filter" hardware packet count, and clearing one rule's counter disturbed the neighbouring rule. Split the two cases on the rule ID / route counter boundary: address PIE rule counters directly at entry = id, word 1, and keep the paired counter/2 read-modify-write for the route counters. Verified on an RTL9302C (Zyxel XGS1210-12) with several concurrent skip_sw flower rules: each now reports its own increasing packet count. Assisted-by: Claude Code (Anthropic Claude Sonnet 5) Signed-off-by: Mark Abe <github@mab.wien>
…llbacks
With tc_flow_lock serializing rtldsa_configure_flower(),
rtldsa_delete_flower() and rtldsa_stats_flower(), and a flow only ever
freed (via kfree_rcu) from rtldsa_delete_flower() / rtldsa_tc_cleanup(),
a looked-up flow cannot be freed while a callback holds tc_flow_lock. The
hand-rolled rcu_read_lock() sections around the rhashtable lookups are
therefore redundant:
- rhashtable_lookup_fast() / _insert_fast() / _remove_fast() already
take rcu_read_lock() internally;
- nothing dereferences a looked-up flow outside tc_flow_lock.
Switch configure_flower() from rhashtable_lookup() to the _fast variant
and drop the explicit rcu_read_lock() / rcu_read_unlock() in all three
callbacks. No functional change.
Assisted-by: Claude Code (Anthropic Claude Sonnet 5)
Signed-off-by: Mark Abe <github@mab.wien>
rtl83xx_setup_tc() and its helpers rtl83xx_setup_tc_block_cb() /
rtl83xx_setup_tc_cls_flower() form a TC_SETUP_BLOCK dispatcher meant to
be reached through .ndo_setup_tc -> rteth_setup_tc() -> ds->ops->
port_setup_tc(). Nothing in target/linux/realtek ever sets
port_setup_tc, and rtl83xx_setup_tc() is assigned to no ops struct
either, so rteth_setup_tc() returns -EOPNOTSUPP and this whole path is
dead.
The cls_flower offload runs entirely through the DSA .cls_flower_add /
_del / _stats switch ops, i.e. rtldsa_pie_cls_flower_{add,del,stats}()
-> rtldsa_{configure,delete,stats}_flower(). Keeping the second,
unreachable dispatcher only invites a reader to mistake it for a live
path.
Remove it. With it gone rtldsa_configure_flower() has a single caller,
rtldsa_pie_cls_flower_add(), which is RTL930x-only and always passes a
real ingress port, so the ingress_port >= 0 guard and the
!pie_rule_id_is_log_counter branch (the rtldsa_packet_cntr_alloc() /
log_data path) are dead too - drop them. rtldsa_tc_init() /
rtldsa_tc_cleanup() stay; they run from the probe path.
Reported-by: Gennaro Cimmino <gcimmino@rayonra.net>
Assisted-by: Claude Code (Anthropic Claude Sonnet 5)
Signed-off-by: Mark Abe <github@mab.wien>
|
Split as requested: patches 1-4 and "hold tc_flow_lock in rtldsa_tc_cleanup()"
Rebases cleanly on #25094; content unchanged from the tested 9-commit stack |
Add hardware offload of ingress
tc/cls_flowerrules on RTL930x DSAuser ports, using the switch's existing Packet Inspection Engine, on top
of the port rate-policing path that is already wired to the RTL93xx
cls_flowerops.Changes
rtldsa_cls_flower_{add,del}route every rule that is not a plain portrate limiter to the PIE path; a new
rtldsa_cls_flower_statsopreports per-rule packet counts.
family_id == RTL9300_FAMILY_ID; RTL931x keepsits current behaviour and the RTL83xx SoC code is untouched.
masks up front, and gains generic EtherType matching alongside the
ARP/IPv4/IPv6 fast paths.
pie_rule.ethertype_mis widened tou16;rtl930x_pie_verify_template()refuses an EtherType match when theactive template lacks the field.
rtl83xx_validate_flow_actions()rejects nonsensical actioncombinations (multiple forwards; drop mixed with other actions).
field. On RTL930x the PIE rule ID is also the LOG counter ID, so the
packet counter needs no separate allocation.
rtldsa_93xx_setup()(RTL9300 only)and destroyed RCU-safely in
rtl83xx_sw_remove()and the probe errorpath.
The first commit is a standalone fix:
rtl930x_packet_cntr_clear()wrotea shared LOG table entry back without reading it first, clobbering the
adjacent counter. It is latent today and becomes reachable once the
offload clears counters at runtime.
Testing
Built for
realtek/rtl930x(kernel 6.18);make target/linux/compileis clean with no new warnings.
Flashed and tested on a Zyxel XGS1210-12:
flower protocol 0x8863 ... action drop(PPPoE discovery)in_hw in_hw_count 1flower protocol ip ip_proto udp dst_port 67 ... action trapin_hw in_hw_count 1flower ... action trapon ARP +tc -s filter showSent hardware 0 bytes 3 pkt)tc filter del+tc qdisc del clsactdmesgerrorsNote
rtl83xx_setup_tc()(the older flow-block-cb path) has noport_setup_tcop wired to it and is currently unreachable; it is kept building and now
shares
rtl83xx_tc_init().Discussion / testing feedback: https://forum.openwrt.org/t/tcfilter-luci-app-tcfilter-persistent-tc-ingress-filters-with-hardware-offload-on-realtek-switches/253331