[observer] High-frequency system check runner (1s interval)#48707
Merged
scottopell merged 1 commit intoq-branch-observerfrom Apr 3, 2026
Merged
[observer] High-frequency system check runner (1s interval)#48707scottopell merged 1 commit intoq-branch-observerfrom
scottopell merged 1 commit intoq-branch-observerfrom
Conversation
Contributor
Go Package Import DifferencesBaseline: e5b320d
|
Contributor
Files inventory check summaryFile checks results against ancestor e5b320de: Results for datadog-agent_7.79.0~devel.git.408.b4bf1a9.pipeline.105946945-1_amd64.deb:Detected file changes:
|
Contributor
Static quality checks❌ Please find below the results from static quality gates Error
Gate failure full details
Static quality gates prevent the PR to merge! Successful checksInfo
2 successful checks with minimal change (< 2 KiB)
On-wire sizes (compressed)
|
5 tasks
scottopell
force-pushed
the
sopell/more-frequent-system-checks
branch
2 times, most recently
from
3d095cf to
1f186c2
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1f186c2638
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
rahulkaukuntla
approved these changes
Apr 3, 2026
CelianR
approved these changes
Apr 3, 2026
Contributor
CelianR
left a comment
CelianR
left a comment
There was a problem hiding this comment.
LGTM, thanks for adding telemetry!
scottopell
force-pushed
the
sopell/more-frequent-system-checks
branch
from
1f186c2 to
b4bf1a9
Compare
Contributor
Author
|
@codex re-review please |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b4bf1a92d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+774
to
+776
| if h.dropCounter != nil { | ||
| h.dropCounter.Add(1, h.source) | ||
| } |
There was a problem hiding this comment.
The new observer.channel.dropped telemetry is only incremented in ObserveMetric; dropped logs, traces, and profiles still update only dropCount. If the observer channel is saturated by non-metric traffic (for example log bursts), this counter will significantly under-report back-pressure by source, which defeats the purpose of this new metric for operational debugging. Please increment dropCounter in the drop paths of ObserveLog, ObserveTrace, and ObserveProfile too.
Useful? React with 👍 / 👎.
Comment on lines
+366
to
+370
| // When true, the observer runs system checks (cpu, memory, disk, io, load, network, etc.) | ||
| // at 1-second intervals and feeds them directly into the anomaly detection pipeline. | ||
| // These high-frequency samples are never forwarded to Datadog intake. | ||
| // The normal 15-second system check pipeline is unaffected. | ||
| config.BindEnvAndSetDefault("observer.high_frequency_system_checks.enabled", false) |
There was a problem hiding this comment.
This change adds observer.high_frequency_system_checks.enabled to runtime defaults, but the high-frequency observer section was removed from config_template.yaml without a replacement entry for the new key. That leaves users without generated config/env documentation for enabling this feature, which is a stale-doc regression for a behavior/config change and makes rollout/support harder.
Useful? React with 👍 / 👎.
Introduces a prototype to test whether 1-second system check metrics improve anomaly detection scores in comp/observer compared to the default 15-second cadence. - Adds `comp/observer/impl/hfrunner`: an observer-owned check runner that instantiates system checks (cpu, memory, disk, io, load, network, uptime, filehandles) using the existing factory catalog, runs them on a 1s tick, and routes output directly into the observer pipeline via a purpose-built `observerSender`. These samples never touch the aggregator or forwarder — intake isolation is guaranteed by design. - When `observer.high_frequency_system_checks.enabled: true`, a `systemFilteredHandle` is placed on the `"all-metrics"` pipeline handle to suppress 15s system.* samples from the scorer, so only the higher-resolution 1s stream influences detection. Filtering uses `MetricSource` enum comparison via type assertion rather than string prefix matching. - Adds `observer.channel.dropped` telemetry counter (tagged by source) to surface back-pressure from the observer's internal channel. - Wires runner shutdown via `fx.Lifecycle.OnStop`; `Runner.Stop()` is guarded with `sync.Once` against double-close. Removes the `observer.metrics.high_frequency_interval` mechanism that previously hacked all check intervals globally via `CheckBase.Interval()`. That approach is replaced entirely by the runner above. - `checkbase.go`: `Interval()` simplified to `return c.checkInterval` - `config.go`: `observer.metrics.high_frequency_interval` and `observer.metrics.enabled` config keys removed - `sender.go`: `AggregatingSender` instantiation path removed - `config_template.yaml`: stale `@param` doc block removed - `TestSystemFilteredHandle`: table-driven unit test covering all 8 system check sources (dropped), non-system sources (pass through), `MetricSourceUnknown` (pass through), and no-sourceProvider samples (pass through). - `TestAggregatingSender_SourcePopulated`: regression test verifying all 12 metric methods on `AggregatingSender` stamp the correct `MetricSource` on observer-bound samples. - QA confirmed via proxy-dumper MiTM: `system.cpu.user` reached intake 4 times in 60s (15s cadence), not 60 times. - Observer debug dump confirmed 475 series under `"system-checks-hf"` with 28-29 points each at strict 1s intervals after 30s runtime. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
scottopell
force-pushed
the
sopell/more-frequent-system-checks
branch
from
b4bf1a9 to
b6fefd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
comp/observer/impl/hfrunner: an observer-owned runner that instantiates system checks (cpu, memory, disk, io, load, network, uptime, filehandles) at 1-second intervals and routes their output directly into the observer anomaly detection pipeline. These samples never touch the aggregator or forwarder — intake isolation is guaranteed by design.observer.high_frequency_system_checks.enabled: true, asystemFilteredHandlesuppresses the normal 15s system.* samples from the scorer so only the higher-resolution stream influences detection. Filtering usesMetricSourceenum comparison (type assertion tosourceProvider) rather than string prefix matching.observer.channel.droppedtelemetry counter (tagged bysource) to surface back-pressure from the observer's internal channel.observer.metrics.high_frequency_intervalmechanism that previously hacked all check intervals globally viaCheckBase.Interval().Hypothesis
Does 1s system metric data improve anomaly detection scores vs the default 15s? This PR provides the infrastructure to answer that question — enable the flag, run evals, compare scores.
To enable
Test plan
TestSystemFilteredHandle: table-driven test — all 8 system sources dropped, non-system sources pass through,MetricSourceUnknownpasses through, no-sourceProvidersamples pass throughTestAggregatingSender_SourcePopulated: regression test — all 12 metric methods stamp correctMetricSourceon observer-bound samplessystem.cpu.userhit intake 4×/60s (15s cadence), not 60×"system-checks-hf"with 28–29 points at strict 1s intervals after 30s runtime🤖 Generated with Claude Code