[Backport 7.78.x] [CXP-3401][agent][windows] Fix remote process collector missing from Windows build by dd-octo-sts[bot] · Pull Request #48830 · DataDog/datadog-agent

dd-octo-sts · 2026-04-02T21:13:14Z

Backport 3fc1dc8 from #48819.

Summary

Fixes remote process collector missing from Windows build, restoring language detection
Adds a Windows E2E test for language detection via the remote_process_collector to the existing windowsTestSuite
include cherry-picked commit from [CXP-3401][agent] skip broken e2e windows language detection test + clean up #48833 to skip broken e2e test

What's broken

PR #46219 split the workloadmeta catalog into trivy / !trivy variants but only included remoteprocesscollector in the trivy-gated file (options.go). Since trivy is in LINUX_ONLY_TAGS, Windows always uses the !trivy build (options_nosbom.go) — which is missing the remote process collector. This broke language detection on Windows since March 20. More details in jira ticket: https://datadoghq.atlassian.net/browse/CXP-3401

Evidence

Diff that introduced the regression — options_nosbom.go created without remoteprocesscollector
LINUX_ONLY_TAGS includes trivy — confirms Windows never gets the trivy build
Validated on Windows Server EC2 running agent 7.78.0-rc.5:
- agent workload-list --json returns {"Entities":{}}
- Agent logs show no remote-process-collector among workloadmeta collector candidates
- Config confirms language_detection.enabled: true

Why the test lives in `tests/process/` instead of `tests/language-detection/`

The test is added to the existing windowsTestSuite in tests/process/windows_test.go to reuse the same Windows EC2 instance already provisioned by the new-e2e-process-windows CI job for efficiency rather than logical organization. Placing it in tests/language-detection/ would require a separate Windows CI job and provision an additional Windows instance, adding ~10 min of extra CI time.

Test plan

E2E test confirms regression (fails against 7.78.0-rc.5 with {"Entities":{}}) (details in jira comment: https://datadoghq.atlassian.net/browse/CXP-3401?focusedCommentId=3134649)
tested manually since e2e test is blocked by testing issue

Ran the python script

❯ sshpass -p "<PASSWORD>" ssh -o StrictHostKeyChecking=no <SERVER> 'Set-Content -Path C:\sleep.py -Value "import time; time.sleep(600)"; Start-Process -FilePath "C:\Program Files\Datadog\Datadog Agent\embedded3\python.exe" -ArgumentList "C:\sleep.py" -WindowStyle Hidden; Start-Sleep 60; Get-Process python'

We can see the python language appear

❯ sshpass -p "<PASSWORD>" ssh -o StrictHostKeyChecking=no <SERVER> '& "C:\Program Files\Datadog\Datadog Agent\bin\agent.exe" workload-list --json' | grep -o '"Language":{[^}]*}' | sort | uniq -c | sort -rn
     96 "Language":{}
      1 "Language":{"Name":"python"}

🤖 Generated with Claude Code

agent-platform-auto-pr · 2026-04-02T21:37:57Z

Files inventory check summary

File checks results against ancestor dbbcf5ea:

Results for datadog-agent_7.78.0~rc.6.git.3.7fab1ce.pipeline.106217155-1_amd64.deb:

No change detected

agent-platform-auto-pr · 2026-04-02T21:47:41Z

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor dbbcf5e
📊 Static Quality Gates Dashboard
🔗 SQG Job

Successful checks

Info

	Quality gate	Change	Size (prev → curr → max)
✅	iot_agent_deb_arm64	+4.0 KiB (0.01% increase)	40.304 → 40.308 → 40.920

30 successful checks with minimal change (< 2 KiB)

	Quality gate	Current Size
✅	agent_deb_amd64	752.488 MiB
✅	agent_deb_amd64_fips	709.361 MiB
✅	agent_heroku_amd64	314.423 MiB
✅	agent_msi	605.985 MiB
✅	agent_rpm_amd64	752.471 MiB
✅	agent_rpm_amd64_fips	709.344 MiB
✅	agent_rpm_arm64	730.875 MiB
✅	agent_rpm_arm64_fips	690.781 MiB
✅	agent_suse_amd64	752.471 MiB
✅	agent_suse_amd64_fips	709.344 MiB
✅	agent_suse_arm64	730.875 MiB
✅	agent_suse_arm64_fips	690.781 MiB
✅	docker_agent_amd64	812.795 MiB
✅	docker_agent_arm64	815.966 MiB
✅	docker_agent_jmx_amd64	1003.711 MiB
✅	docker_agent_jmx_arm64	995.660 MiB
✅	docker_cluster_agent_amd64	205.403 MiB
✅	docker_cluster_agent_arm64	219.741 MiB
✅	docker_cws_instrumentation_amd64	7.142 MiB
✅	docker_cws_instrumentation_arm64	6.689 MiB
✅	docker_dogstatsd_amd64	39.230 MiB
✅	docker_dogstatsd_arm64	37.445 MiB
✅	dogstatsd_deb_amd64	29.874 MiB
✅	dogstatsd_deb_arm64	28.027 MiB
✅	dogstatsd_rpm_amd64	29.874 MiB
✅	dogstatsd_suse_amd64	29.874 MiB
✅	iot_agent_deb_amd64	43.257 MiB
✅	iot_agent_deb_armhf	41.052 MiB
✅	iot_agent_rpm_amd64	43.258 MiB
✅	iot_agent_suse_amd64	43.258 MiB

On-wire sizes (compressed)

	Quality gate	Change	Size (prev → curr → max)
✅	agent_deb_amd64	-18.38 KiB (0.01% reduction)	174.733 → 174.715 → 178.360
✅	agent_deb_amd64_fips	neutral	165.263 MiB → 172.790
✅	agent_heroku_amd64	+6.11 KiB (0.01% increase)	75.065 → 75.071 → 79.970
✅	agent_msi	-4.0 KiB (0.00% reduction)	138.406 → 138.402 → 146.220
✅	agent_rpm_amd64	+6.02 KiB (0.00% increase)	177.311 → 177.317 → 181.830
✅	agent_rpm_amd64_fips	-20.09 KiB (0.01% reduction)	167.419 → 167.399 → 173.370
✅	agent_rpm_arm64	+44.7 KiB (0.03% increase)	159.311 → 159.355 → 163.060
✅	agent_rpm_arm64_fips	+20.14 KiB (0.01% increase)	151.369 → 151.389 → 156.170
✅	agent_suse_amd64	+6.02 KiB (0.00% increase)	177.311 → 177.317 → 181.830
✅	agent_suse_amd64_fips	-20.09 KiB (0.01% reduction)	167.419 → 167.399 → 173.370
✅	agent_suse_arm64	+44.7 KiB (0.03% increase)	159.311 → 159.355 → 163.060
✅	agent_suse_arm64_fips	+20.14 KiB (0.01% increase)	151.369 → 151.389 → 156.170
✅	docker_agent_amd64	neutral	267.995 MiB → 272.480
✅	docker_agent_arm64	neutral	255.198 MiB → 261.060
✅	docker_agent_jmx_amd64	-5.05 KiB (0.00% reduction)	336.649 → 336.644 → 341.100
✅	docker_agent_jmx_arm64	-2.55 KiB (0.00% reduction)	319.831 → 319.828 → 325.620
✅	docker_cluster_agent_amd64	neutral	71.963 MiB → 72.920
✅	docker_cluster_agent_arm64	neutral	67.536 MiB → 68.220
✅	docker_cws_instrumentation_amd64	neutral	2.999 MiB → 3.330
✅	docker_cws_instrumentation_arm64	neutral	2.729 MiB → 3.090
✅	docker_dogstatsd_amd64	neutral	15.171 MiB → 15.820
✅	docker_dogstatsd_arm64	neutral	14.484 MiB → 14.830
✅	dogstatsd_deb_amd64	+2.16 KiB (0.03% increase)	7.889 → 7.891 → 8.790
✅	dogstatsd_deb_arm64	neutral	6.775 MiB → 7.710
✅	dogstatsd_rpm_amd64	neutral	7.900 MiB → 8.800
✅	dogstatsd_suse_amd64	neutral	7.900 MiB → 8.800
✅	iot_agent_deb_amd64	neutral	11.395 MiB → 12.040
✅	iot_agent_deb_arm64	neutral	9.698 MiB → 10.450
✅	iot_agent_deb_armhf	neutral	9.936 MiB → 10.620
✅	iot_agent_rpm_amd64	neutral	11.411 MiB → 12.060
✅	iot_agent_suse_amd64	neutral	11.411 MiB → 12.060

cit-pr-commenter-54b7da · 2026-04-02T22:02:41Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 920ec363-3cab-44db-ab9b-bb4e6b9e7cae

Baseline: dbbcf5e
Comparison: 7fab1ce
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
❌	docker_containers_cpu	% cpu utilization	+6.36	[+3.25, +9.47]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
❌	docker_containers_cpu	% cpu utilization	+6.36	[+3.25, +9.47]	1	Logs
➖	quality_gate_logs	% cpu utilization	+1.71	[+0.10, +3.33]	1	Logs bounds checks dashboard
➖	quality_gate_metrics_logs	memory utilization	+0.87	[+0.63, +1.10]	1	Logs bounds checks dashboard
➖	tcp_syslog_to_blackhole	ingress throughput	+0.46	[+0.31, +0.61]	1	Logs
➖	docker_containers_memory	memory utilization	+0.45	[+0.38, +0.53]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.27	[+0.21, +0.32]	1	Logs
➖	ddot_metrics_sum_cumulative	memory utilization	+0.26	[+0.12, +0.40]	1	Logs
➖	otlp_ingest_logs	memory utilization	+0.22	[+0.12, +0.32]	1	Logs
➖	quality_gate_idle	memory utilization	+0.19	[+0.14, +0.24]	1	Logs bounds checks dashboard
➖	file_tree	memory utilization	+0.13	[+0.08, +0.19]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.11	[+0.07, +0.15]	1	Logs bounds checks dashboard
➖	file_to_blackhole_500ms_latency	egress throughput	+0.09	[-0.30, +0.49]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	+0.08	[-0.43, +0.59]	1	Logs
➖	otlp_ingest_metrics	memory utilization	+0.07	[-0.09, +0.23]	1	Logs
➖	ddot_metrics	memory utilization	+0.07	[-0.12, +0.26]	1	Logs
➖	ddot_logs	memory utilization	+0.02	[-0.04, +0.09]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.02	[-0.41, +0.45]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.01	[-0.19, +0.21]	1	Logs
➖	uds_dogstatsd_to_api_v3	ingress throughput	+0.01	[-0.19, +0.21]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.10, +0.11]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.02	[-0.11, +0.06]	1	Logs
➖	ddot_metrics_sum_cumulativetodelta_exporter	memory utilization	-0.04	[-0.26, +0.19]	1	Logs
➖	ddot_metrics_sum_delta	memory utilization	-0.22	[-0.39, -0.05]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	observed_value	links
✅	docker_containers_cpu	simple_check_run	10/10	680 ≥ 26
✅	docker_containers_memory	memory_usage	10/10	276.41MiB ≤ 370MiB
✅	docker_containers_memory	simple_check_run	10/10	704 ≥ 26
✅	file_to_blackhole_0ms_latency	memory_usage	10/10	0.19GiB ≤ 1.20GiB
✅	file_to_blackhole_0ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10	0.23GiB ≤ 1.20GiB
✅	file_to_blackhole_1000ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_100ms_latency	memory_usage	10/10	0.19GiB ≤ 1.20GiB
✅	file_to_blackhole_100ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_500ms_latency	memory_usage	10/10	0.22GiB ≤ 1.20GiB
✅	file_to_blackhole_500ms_latency	missed_bytes	10/10	0B = 0B
✅	quality_gate_idle	intake_connections	10/10	3 = 3	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	173.67MiB ≤ 175MiB	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	2 ≤ 3	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	490.91MiB ≤ 550MiB	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	3 ≤ 6	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	205.43MiB ≤ 220MiB	bounds checks dashboard
✅	quality_gate_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	368.41 ≤ 2000	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	3 ≤ 6	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	415.64MiB ≤ 475MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.

gengnamstyle · 2026-04-06T16:15:24Z

cherry picked commit from #48833 to clean up e2e test

…Windows build (#48819) ## Summary - Fixes remote process collector missing from Windows build, restoring language detection - Adds a Windows E2E test for language detection via the `remote_process_collector` to the existing `windowsTestSuite` ## What's broken PR #46219 split the workloadmeta catalog into `trivy` / `!trivy` variants but only included `remoteprocesscollector` in the `trivy`-gated file (`options.go`). Since `trivy` is in `LINUX_ONLY_TAGS`, Windows always uses the `!trivy` build (`options_nosbom.go`) — which is missing the remote process collector. This broke language detection on Windows since March 20. More details in jira ticket: https://datadoghq.atlassian.net/browse/CXP-3401 ## Evidence - [Diff that introduced the regression](20aa50f) — `options_nosbom.go` created without `remoteprocesscollector` - [LINUX_ONLY_TAGS includes trivy](https://github.com/DataDog/datadog-agent/blob/main/tasks/build_tags.py) — confirms Windows never gets the `trivy` build - Validated on Windows Server EC2 running agent 7.78.0-rc.5: - `agent workload-list --json` returns `{"Entities":{}}` - Agent logs show no `remote-process-collector` among workloadmeta collector candidates - Config confirms `language_detection.enabled: true` ## Why the test lives in `tests/process/` instead of `tests/language-detection/` The test is added to the existing `windowsTestSuite` in `tests/process/windows_test.go` to reuse the same Windows EC2 instance already provisioned by the `new-e2e-process-windows` CI job for efficiency rather than logical organization. Placing it in `tests/language-detection/` would require a separate Windows CI job and provision an additional Windows instance, adding ~10 min of extra CI time. ## Test plan - [x] CI passes - [x] E2E test confirms regression (fails against 7.78.0-rc.5 with `{"Entities":{}}`) (details in jira comment: https://datadoghq.atlassian.net/browse/CXP-3401?focusedCommentId=3134649) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: matthew.geng <matthew.geng@datadoghq.com> (cherry picked from commit 3fc1dc8) ___ Co-authored-by: Matthew Geng <matthew.geng@datadoghq.com>

Remove PID matching from TestLanguageDetectionWindows and check for any process with language=python instead. The SSH session wraps commands in PowerShell, causing the PID from Get-CimInstance to differ from the one in workloadmeta, leading to false failures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

AliDatadog · 2026-04-07T08:03:04Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-04-07T08:03:09Z

View all feedbacks in Devflow UI.

2026-04-07 08:03:08 UTC ℹ️ Start processing command /merge

2026-04-07 08:03:13 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in 7.78.x is approximately 1h (p90).

2026-04-07 08:45:52 UTC ℹ️ MergeQueue: This merge request was merged

dd-octo-sts bot requested review from a team as code owners April 2, 2026 21:13

dd-octo-sts bot and others added 2 commits April 6, 2026 12:21

gengnamstyle force-pushed the backport-48819-to-7.78.x branch from 5b0fb02 to 7fab1ce Compare April 6, 2026 16:21

robertjli approved these changes Apr 6, 2026

View reviewed changes

gengnamstyle self-assigned this Apr 6, 2026

AliDatadog approved these changes Apr 7, 2026

View reviewed changes

gh-worker-dd-mergequeue-cf854d bot merged commit de0a33a into 7.78.x Apr 7, 2026
295 checks passed

gh-worker-dd-mergequeue-cf854d bot deleted the backport-48819-to-7.78.x branch April 7, 2026 08:45

github-actions bot added this to the 7.78.0 milestone Apr 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 7.78.x] [CXP-3401][agent][windows] Fix remote process collector missing from Windows build#48830

[Backport 7.78.x] [CXP-3401][agent][windows] Fix remote process collector missing from Windows build#48830
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into7.78.xfrom
backport-48819-to-7.78.x

dd-octo-sts bot commented Apr 2, 2026 •

edited by gengnamstyle

Loading

Uh oh!

agent-platform-auto-pr bot commented Apr 2, 2026 •

edited

Loading

Uh oh!

agent-platform-auto-pr bot commented Apr 2, 2026 •

edited

Loading

Info

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 2, 2026 •

edited

Loading

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

Uh oh!

gengnamstyle commented Apr 6, 2026

Uh oh!

AliDatadog commented Apr 7, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dd-octo-sts bot commented Apr 2, 2026 • edited by gengnamstyle Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's broken

Evidence

Why the test lives in tests/process/ instead of tests/language-detection/

Test plan

Uh oh!

agent-platform-auto-pr bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Files inventory check summary

Results for datadog-agent_7.78.0~rc.6.git.3.7fab1ce.pipeline.106217155-1_amd64.deb:

Uh oh!

agent-platform-auto-pr bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Static quality checks

Info

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

gengnamstyle commented Apr 6, 2026

Uh oh!

AliDatadog commented Apr 7, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dd-octo-sts bot commented Apr 2, 2026 •

edited by gengnamstyle

Loading

Why the test lives in `tests/process/` instead of `tests/language-detection/`

agent-platform-auto-pr bot commented Apr 2, 2026 •

edited

Loading

agent-platform-auto-pr bot commented Apr 2, 2026 •

edited

Loading

cit-pr-commenter-54b7da bot commented Apr 2, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Apr 7, 2026 •

edited

Loading