Skip to content

[RFC] Introduce log ratelimit#620

Open
vlrpl wants to merge 2 commits into
retis-org:mainfrom
vlrpl:log-ratelimit
Open

[RFC] Introduce log ratelimit#620
vlrpl wants to merge 2 commits into
retis-org:mainfrom
vlrpl:log-ratelimit

Conversation

@vlrpl

@vlrpl vlrpl commented Apr 3, 2026

Copy link
Copy Markdown
Contributor

The idea is to avoid infinite printouts due to recurring error condition (e.g. unexpected corruption).

Requires 5.12 support (mostly for __sync_val_compare_and_swap).
As of today we roughly support 5.8+ (not really verified, only statically inspected). If that really matters, we can have something like a probe program for BPF_ATOMIC | BPF_CMPXCHG and instruct the probes to fall back to the non-ratelimited flavors.

vlrpl added 2 commits April 3, 2026 18:09
Add rate_limit() and log_*_rl() macros to guard hotpath log
calls. A shared BPF_MAP_TYPE_ARRAY holds the last-allowed timestamp and
a suppressed-message counter, updated via atomic CAS across all CPUs and
probe programs. Due to the lockless design, two CPUs racing at the
interval boundary may both pass the check, causing at most one extra
message to be emitted per interval. Suppressed counts are reported at
shutdown alongside dropped event counters.

Signed-off-by: Paolo Valerio <pvalerio@redhat.com>
Signed-off-by: Paolo Valerio <pvalerio@redhat.com>
@vlrpl vlrpl added the run-functional-tests Request functional tests to be run by CI label Apr 3, 2026
@vlrpl

vlrpl commented Apr 3, 2026

Copy link
Copy Markdown
Contributor Author

CAS requires v3 instructions support (the patch doesn't have it). Didn't check the actual reason yet (i.e. no full nor partial v3 support) , but clang 14 still seem to crash despite forcing v3.

@atenart

atenart commented Apr 17, 2026

Copy link
Copy Markdown
Member

Did you experience a flooding of eBPF log messages while using Retis? What was causing it?

@vlrpl

vlrpl commented Apr 22, 2026

Copy link
Copy Markdown
Contributor Author

Did you experience a flooding of eBPF log messages while using Retis? What was causing it?

It was observed in some reports (e.g. ovs-related, WRT flow enrichment, IIRC).
In general, this is a potential problem with the hot path and unexpectedly repeated error conditions (although it might not be that severe, as repeated error conditions like that will likely make the event generation unreliable anyways).

@atenart

atenart commented Apr 22, 2026

Copy link
Copy Markdown
Member

As discussed offline, a different approach could be used too with _once variants of the error helpers (we can even make the "once" per-program to not avoid needing shared variables/maps). This has the advantage of not relying on a specific version (as described above).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-functional-tests Request functional tests to be run by CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants