attempt to reduce flakiness of trace exporter shutdown tests #1128

ekump · 2025-07-03T20:13:07Z

What does this PR do?

Attempts to reduce the likelihood of the trace exporter shutdown unit tests flaking on CI by:

Directly checking AGENT_INFO_CACHE to be Some instead of checking the mock /info endpoint is hit. This is a potential race condition in tests as there is a non-zero amount of time between the endpoint being hit, the cache being populated, and the exporter checking the cache to know if it should enable the stats worker.
Added a test-only function to check if stats computation is enabled and the worker is running. In tests it may be possible that we start the shutdown before stats worker is active.

Motivation

Flaky tests aren't cool

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

pr-commenter · 2025-07-22T20:23:12Z

Benchmarks

Comparison

Benchmark execution time: 2025-07-22 20:40:48

Comparing candidate commit e5ff9fe in PR branch ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test with baseline commit 4eb2b86 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 2 unstable metrics.

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	206.010µs	206.625µs ± 0.551µs	206.492µs ± 0.193µs	206.691µs	207.790µs	208.009µs	210.964µs	2.17%	3.254	18.491	0.27%	0.039µs	1	200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	4740148.274op/s	4839724.109op/s ± 12802.750op/s	4842791.851op/s ± 4519.162op/s	4847296.090op/s	4851046.392op/s	4853076.926op/s	4854125.016op/s	0.23%	-3.177	17.624	0.26%	905.291op/s	1	200
normalization/normalize_name/normalize_name/bad-name	execution_time	18.224µs	18.289µs ± 0.064µs	18.272µs ± 0.021µs	18.299µs	18.420µs	18.528µs	18.671µs	2.18%	2.876	9.976	0.35%	0.005µs	1	200
normalization/normalize_name/normalize_name/bad-name	throughput	53558609.546op/s	54679202.680op/s ± 189100.673op/s	54728294.835op/s ± 63879.382op/s	54782986.334op/s	54834815.648op/s	54868197.414op/s	54873833.989op/s	0.27%	-2.838	9.692	0.34%	13371.437op/s	1	200
normalization/normalize_name/normalize_name/good	execution_time	10.943µs	11.010µs ± 0.035µs	11.008µs ± 0.023µs	11.032µs	11.067µs	11.106µs	11.120µs	1.02%	0.421	0.036	0.32%	0.002µs	1	200
normalization/normalize_name/normalize_name/good	throughput	89927016.051op/s	90827333.570op/s ± 290641.180op/s	90841741.685op/s ± 190837.200op/s	91022806.736op/s	91269543.727op/s	91366015.373op/s	91382176.183op/s	0.59%	-0.404	0.009	0.32%	20551.435op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	[206.548µs; 206.701µs] or [-0.037%; +0.037%]	None	None	None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	[4837949.771op/s; 4841498.447op/s] or [-0.037%; +0.037%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	execution_time	[18.280µs; 18.298µs] or [-0.048%; +0.048%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	throughput	[54652995.145op/s; 54705410.214op/s] or [-0.048%; +0.048%]	None	None	None
normalization/normalize_name/normalize_name/good	execution_time	[11.005µs; 11.015µs] or [-0.044%; +0.044%]	None	None	None
normalization/normalize_name/normalize_name/good	throughput	[90787053.497op/s; 90867613.642op/s] or [-0.044%; +0.044%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	17.905µs	26.318µs ± 10.104µs	18.451µs ± 0.423µs	34.742µs	44.424µs	53.120µs	76.305µs	313.56%	1.114	1.684	38.30%	0.714µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[24.918µs; 27.719µs] or [-5.321%; +5.321%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
credit_card/is_card_number/	execution_time	3.892µs	3.911µs ± 0.002µs	3.911µs ± 0.001µs	3.913µs	3.915µs	3.917µs	3.917µs	0.15%	-2.197	17.726	0.06%	0.000µs	1	200
credit_card/is_card_number/	throughput	255275842.113op/s	255662829.493op/s ± 158282.796op/s	255659898.378op/s ± 92659.523op/s	255754840.089op/s	255869657.958op/s	255938676.578op/s	256910805.249op/s	0.49%	2.225	17.981	0.06%	11192.284op/s	1	200
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	78.942µs	80.367µs ± 0.713µs	80.317µs ± 0.495µs	80.868µs	81.612µs	82.025µs	82.481µs	2.69%	0.346	-0.301	0.88%	0.050µs	1	200
credit_card/is_card_number/ 3782-8224-6310-005	throughput	12124025.483op/s	12443872.820op/s ± 110010.407op/s	12450592.321op/s ± 76685.129op/s	12519621.707op/s	12611680.225op/s	12657193.991op/s	12667490.831op/s	1.74%	-0.305	-0.344	0.88%	7778.910op/s	1	200
credit_card/is_card_number/ 378282246310005	execution_time	74.333µs	74.910µs ± 0.182µs	74.894µs ± 0.095µs	74.985µs	75.226µs	75.482µs	75.608µs	0.95%	0.677	2.123	0.24%	0.013µs	1	200
credit_card/is_card_number/ 378282246310005	throughput	13226125.388op/s	13349457.285op/s ± 32352.189op/s	13352271.643op/s ± 16960.018op/s	13369381.443op/s	13394138.773op/s	13422170.411op/s	13453019.696op/s	0.75%	-0.651	2.088	0.24%	2287.645op/s	1	200
credit_card/is_card_number/37828224631	execution_time	3.894µs	3.911µs ± 0.003µs	3.911µs ± 0.002µs	3.913µs	3.917µs	3.920µs	3.922µs	0.27%	-0.005	3.777	0.08%	0.000µs	1	200
credit_card/is_card_number/37828224631	throughput	254988028.731op/s	255659264.903op/s ± 208807.491op/s	255670296.548op/s ± 119549.515op/s	255804310.905op/s	255903054.475op/s	255972394.783op/s	256772861.399op/s	0.43%	0.019	3.830	0.08%	14764.919op/s	1	200
credit_card/is_card_number/378282246310005	execution_time	71.563µs	72.089µs ± 0.184µs	72.075µs ± 0.108µs	72.194µs	72.430µs	72.500µs	72.647µs	0.79%	0.100	0.483	0.25%	0.013µs	1	200
credit_card/is_card_number/378282246310005	throughput	13765250.585op/s	13871847.931op/s ± 35327.954op/s	13874483.438op/s ± 20791.477op/s	13892817.780op/s	13925166.684op/s	13956561.973op/s	13973652.327op/s	0.71%	-0.081	0.487	0.25%	2498.064op/s	1	200
credit_card/is_card_number/37828224631000521389798	execution_time	52.125µs	52.189µs ± 0.030µs	52.185µs ± 0.020µs	52.209µs	52.240µs	52.265µs	52.291µs	0.20%	0.473	-0.009	0.06%	0.002µs	1	200
credit_card/is_card_number/37828224631000521389798	throughput	19123886.930op/s	19161164.678op/s ± 10988.237op/s	19162667.292op/s ± 7382.431op/s	19169175.487op/s	19177028.700op/s	19181586.905op/s	19184529.595op/s	0.11%	-0.470	-0.014	0.06%	776.986op/s	1	200
credit_card/is_card_number/x371413321323331	execution_time	5.691µs	5.896µs ± 0.072µs	5.896µs ± 0.052µs	5.957µs	6.003µs	6.034µs	6.038µs	2.40%	-0.208	-0.334	1.22%	0.005µs	1	200
credit_card/is_card_number/x371413321323331	throughput	165631413.264op/s	169639847.722op/s ± 2087307.281op/s	169601976.435op/s ± 1506604.244op/s	170847528.617op/s	173008556.075op/s	174849086.213op/s	175725654.575op/s	3.61%	0.269	-0.259	1.23%	147594.913op/s	1	200
credit_card/is_card_number_no_luhn/	execution_time	3.893µs	3.912µs ± 0.003µs	3.913µs ± 0.002µs	3.914µs	3.916µs	3.919µs	3.920µs	0.17%	-1.410	8.664	0.07%	0.000µs	1	200
credit_card/is_card_number_no_luhn/	throughput	255132508.761op/s	255602649.115op/s ± 187302.274op/s	255576888.628op/s ± 112605.509op/s	255727912.704op/s	255864889.583op/s	255942690.359op/s	256857322.036op/s	0.50%	1.429	8.806	0.07%	13244.271op/s	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	64.160µs	64.330µs ± 0.097µs	64.321µs ± 0.050µs	64.367µs	64.494µs	64.693µs	64.808µs	0.76%	1.721	4.876	0.15%	0.007µs	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	15430086.968op/s	15544773.627op/s ± 23282.079op/s	15547070.000op/s ± 12001.933op/s	15560328.553op/s	15572271.548op/s	15583004.617op/s	15586028.684op/s	0.25%	-1.704	4.791	0.15%	1646.292op/s	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	57.362µs	57.566µs ± 0.111µs	57.542µs ± 0.066µs	57.629µs	57.766µs	57.863µs	57.993µs	0.78%	0.873	0.813	0.19%	0.008µs	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	17243562.948op/s	17371383.956op/s ± 33452.508op/s	17378497.585op/s ± 19852.305op/s	17395576.740op/s	17416201.248op/s	17426778.301op/s	17433280.132op/s	0.32%	-0.862	0.778	0.19%	2365.450op/s	1	200
credit_card/is_card_number_no_luhn/37828224631	execution_time	3.892µs	3.913µs ± 0.003µs	3.913µs ± 0.002µs	3.916µs	3.918µs	3.920µs	3.920µs	0.17%	-1.291	7.038	0.08%	0.000µs	1	200
credit_card/is_card_number_no_luhn/37828224631	throughput	255110643.392op/s	255542841.436op/s ± 216919.961op/s	255534030.647op/s ± 147683.389op/s	255682741.415op/s	255884143.186op/s	255973525.902op/s	256936610.211op/s	0.55%	1.309	7.175	0.08%	15338.558op/s	1	200
credit_card/is_card_number_no_luhn/378282246310005	execution_time	54.577µs	54.854µs ± 0.170µs	54.821µs ± 0.093µs	54.931µs	55.184µs	55.475µs	55.513µs	1.26%	1.418	2.480	0.31%	0.012µs	1	200
credit_card/is_card_number_no_luhn/378282246310005	throughput	18013832.261op/s	18230249.126op/s ± 56320.723op/s	18241244.054op/s ± 31067.215op/s	18270163.910op/s	18297895.379op/s	18312704.293op/s	18322635.784op/s	0.45%	-1.395	2.392	0.31%	3982.476op/s	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	52.131µs	52.202µs ± 0.042µs	52.201µs ± 0.022µs	52.219µs	52.264µs	52.299µs	52.533µs	0.64%	2.789	18.631	0.08%	0.003µs	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	19035717.947op/s	19156218.334op/s ± 15248.292op/s	19156723.048op/s ± 8240.126op/s	19166594.542op/s	19175334.480op/s	19178742.781op/s	19182605.214op/s	0.14%	-2.758	18.321	0.08%	1078.217op/s	1	200
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	5.732µs	5.881µs ± 0.065µs	5.878µs ± 0.050µs	5.929µs	5.991µs	6.003µs	6.037µs	2.71%	0.137	-0.712	1.11%	0.005µs	1	200
credit_card/is_card_number_no_luhn/x371413321323331	throughput	165632907.923op/s	170049781.167op/s ± 1889919.805op/s	170126955.598op/s ± 1446366.686op/s	171509349.008op/s	172982575.202op/s	173711877.884op/s	174468300.661op/s	2.55%	-0.095	-0.717	1.11%	133637.511op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
credit_card/is_card_number/	execution_time	[3.911µs; 3.912µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/	throughput	[255640893.020op/s; 255684765.967op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	[80.268µs; 80.466µs] or [-0.123%; +0.123%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	throughput	[12428626.435op/s; 12459119.204op/s] or [-0.123%; +0.123%]	None	None	None
credit_card/is_card_number/ 378282246310005	execution_time	[74.885µs; 74.935µs] or [-0.034%; +0.034%]	None	None	None
credit_card/is_card_number/ 378282246310005	throughput	[13344973.583op/s; 13353940.987op/s] or [-0.034%; +0.034%]	None	None	None
credit_card/is_card_number/37828224631	execution_time	[3.911µs; 3.912µs] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number/37828224631	throughput	[255630326.193op/s; 255688203.613op/s] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number/378282246310005	execution_time	[72.063µs; 72.114µs] or [-0.035%; +0.035%]	None	None	None
credit_card/is_card_number/378282246310005	throughput	[13866951.816op/s; 13876744.046op/s] or [-0.035%; +0.035%]	None	None	None
credit_card/is_card_number/37828224631000521389798	execution_time	[52.185µs; 52.193µs] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number/37828224631000521389798	throughput	[19159641.814op/s; 19162687.542op/s] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number/x371413321323331	execution_time	[5.886µs; 5.906µs] or [-0.170%; +0.170%]	None	None	None
credit_card/is_card_number/x371413321323331	throughput	[169350567.008op/s; 169929128.437op/s] or [-0.171%; +0.171%]	None	None	None
credit_card/is_card_number_no_luhn/	execution_time	[3.912µs; 3.913µs] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/	throughput	[255576690.821op/s; 255628607.409op/s] or [-0.010%; +0.010%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	[64.317µs; 64.344µs] or [-0.021%; +0.021%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	[15541546.955op/s; 15548000.299op/s] or [-0.021%; +0.021%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	[57.551µs; 57.582µs] or [-0.027%; +0.027%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	[17366747.760op/s; 17376020.152op/s] or [-0.027%; +0.027%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	execution_time	[3.913µs; 3.914µs] or [-0.012%; +0.012%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	throughput	[255512778.416op/s; 255572904.457op/s] or [-0.012%; +0.012%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	execution_time	[54.831µs; 54.878µs] or [-0.043%; +0.043%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	throughput	[18222443.615op/s; 18238054.637op/s] or [-0.043%; +0.043%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	[52.197µs; 52.208µs] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	[19154105.068op/s; 19158331.601op/s] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	[5.872µs; 5.890µs] or [-0.154%; +0.154%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	throughput	[169787856.459op/s; 170311705.876op/s] or [-0.154%; +0.154%]	None	None	None

Group 4

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching string interning on wordpress profile	execution_time	143.608µs	144.066µs ± 0.638µs	143.985µs ± 0.149µs	144.137µs	144.534µs	145.500µs	152.071µs	5.62%	10.113	122.826	0.44%	0.045µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching string interning on wordpress profile	execution_time	[143.978µs; 144.155µs] or [-0.061%; +0.061%]	None	None	None

Group 5

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	85.819µs	85.975µs ± 0.170µs	85.940µs ± 0.043µs	86.001µs	86.159µs	86.361µs	87.915µs	2.30%	7.825	83.911	0.20%	0.012µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[85.951µs; 85.998µs] or [-0.027%; +0.027%]	None	None	None

Group 6

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.191µs	3.192µs ± 1.390µs	2.990µs ± 0.031µs	3.015µs	3.652µs	13.651µs	14.750µs	393.33%	7.329	55.046	43.44%	0.098µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[2.999µs; 3.384µs] or [-6.036%; +6.036%]	None	None	None

Group 7

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	535.575µs	536.525µs ± 0.853µs	536.465µs ± 0.303µs	536.756µs	537.225µs	537.835µs	546.861µs	1.94%	9.001	106.250	0.16%	0.060µs	1	200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	1828619.225op/s	1863851.854op/s ± 2920.871op/s	1864052.972op/s ± 1053.259op/s	1865136.326op/s	1866216.023op/s	1866777.593op/s	1867150.691op/s	0.17%	-8.872	104.153	0.16%	206.537op/s	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	379.364µs	380.108µs ± 0.328µs	380.098µs ± 0.221µs	380.340µs	380.652µs	380.862µs	380.973µs	0.23%	0.073	-0.270	0.09%	0.023µs	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	2624860.026op/s	2630832.749op/s ± 2271.323op/s	2630897.593op/s ± 1528.188op/s	2632226.087op/s	2634691.229op/s	2635778.626op/s	2635992.274op/s	0.19%	-0.069	-0.272	0.09%	160.607op/s	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	195.038µs	195.647µs ± 0.577µs	195.550µs ± 0.238µs	195.857µs	196.100µs	196.274µs	200.974µs	2.77%	6.604	55.394	0.29%	0.041µs	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	4975777.783op/s	5111282.158op/s ± 14786.131op/s	5113790.590op/s ± 6236.740op/s	5118672.550op/s	5122560.323op/s	5123181.687op/s	5127203.213op/s	0.26%	-6.483	53.965	0.29%	1045.537op/s	1	200
normalization/normalize_service/normalize_service/[empty string]	execution_time	36.812µs	37.009µs ± 0.114µs	37.006µs ± 0.095µs	37.094µs	37.195µs	37.262µs	37.282µs	0.75%	0.244	-0.823	0.31%	0.008µs	1	200
normalization/normalize_service/normalize_service/[empty string]	throughput	26822899.412op/s	27020633.947op/s ± 83127.911op/s	27022955.764op/s ± 69287.033op/s	27092415.375op/s	27142910.575op/s	27154965.527op/s	27165173.471op/s	0.53%	-0.233	-0.832	0.31%	5878.031op/s	1	200
normalization/normalize_service/normalize_service/test_ASCII	execution_time	45.895µs	46.152µs ± 0.139µs	46.166µs ± 0.057µs	46.205µs	46.273µs	46.326µs	47.626µs	3.16%	5.803	61.563	0.30%	0.010µs	1	200
normalization/normalize_service/normalize_service/test_ASCII	throughput	20996860.969op/s	21667610.701op/s ± 64124.836op/s	21661066.882op/s ± 26693.323op/s	21701657.123op/s	21746127.287op/s	21770788.771op/s	21788828.964op/s	0.59%	-5.539	58.052	0.30%	4534.311op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	[536.406µs; 536.643µs] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	[1863447.050op/s; 1864256.659op/s] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	[380.063µs; 380.154µs] or [-0.012%; +0.012%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	[2630517.966op/s; 2631147.533op/s] or [-0.012%; +0.012%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	[195.567µs; 195.727µs] or [-0.041%; +0.041%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	[5109232.943op/s; 5113331.374op/s] or [-0.040%; +0.040%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	execution_time	[36.993µs; 37.025µs] or [-0.043%; +0.043%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	throughput	[27009113.218op/s; 27032154.676op/s] or [-0.043%; +0.043%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	execution_time	[46.133µs; 46.171µs] or [-0.042%; +0.042%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	throughput	[21658723.616op/s; 21676497.787op/s] or [-0.041%; +0.041%]	None	None	None

Group 8

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
redis/obfuscate_redis_string	execution_time	33.371µs	33.896µs ± 0.863µs	33.496µs ± 0.043µs	33.591µs	35.725µs	35.816µs	37.069µs	10.66%	1.738	1.282	2.54%	0.061µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
redis/obfuscate_redis_string	execution_time	[33.776µs; 34.016µs] or [-0.353%; +0.353%]	None	None	None

Group 9

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
tags/replace_trace_tags	execution_time	2.432µs	2.461µs ± 0.031µs	2.453µs ± 0.011µs	2.465µs	2.550µs	2.561µs	2.571µs	4.83%	2.224	4.110	1.27%	0.002µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
tags/replace_trace_tags	execution_time	[2.457µs; 2.466µs] or [-0.176%; +0.176%]	None	None	None

Group 10

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching deserializing traces from msgpack to their internal representation	execution_time	73.364ms	73.604ms ± 0.147ms	73.559ms ± 0.085ms	73.693ms	73.903ms	74.017ms	74.121ms	0.77%	0.950	0.578	0.20%	0.010ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching deserializing traces from msgpack to their internal representation	execution_time	[73.584ms; 73.625ms] or [-0.028%; +0.028%]	None	None	None

Group 11

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_trace/test_trace	execution_time	245.137ns	255.611ns ± 13.217ns	249.977ns ± 2.915ns	258.139ns	287.080ns	300.376ns	302.042ns	20.83%	1.900	2.783	5.16%	0.935ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_trace/test_trace	execution_time	[253.779ns; 257.442ns] or [-0.717%; +0.717%]	None	None	None

Group 12

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
ip_address/quantize_peer_ip_address_benchmark	execution_time	5.038µs	5.117µs ± 0.043µs	5.123µs ± 0.037µs	5.154µs	5.177µs	5.180µs	5.181µs	1.13%	-0.174	-1.412	0.83%	0.003µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark	execution_time	[5.111µs; 5.123µs] or [-0.115%; +0.115%]	None	None	None

Group 13

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e5ff9fe`	1753214584	ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	8.213ms	8.230ms ± 0.009ms	8.228ms ± 0.006ms	8.234ms	8.245ms	8.252ms	8.279ms	0.62%	1.280	3.944	0.11%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[8.228ms; 8.231ms] or [-0.015%; +0.015%]	None	None	None

Baseline

Omitted due to size.

codecov-commenter · 2025-07-22T20:23:35Z

Codecov Report

Attention: Patch coverage is 58.06452% with 13 lines in your changes missing coverage. Please review.

Project coverage is 71.27%. Comparing base (4eb2b86) to head (3e8a609).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1128      +/-   ##
==========================================
- Coverage   71.30%   71.27%   -0.03%     
==========================================
  Files         345      345              
  Lines       53148    53174      +26     
==========================================
+ Hits        37898    37901       +3     
- Misses      15250    15273      +23

Components	Coverage Δ
datadog-crashtracker	`43.89% <ø> (-0.06%)`	⬇️
datadog-crashtracker-ffi	`5.93% <ø> (ø)`
datadog-alloc	`98.73% <ø> (ø)`
data-pipeline	`86.83% <58.06%> (-0.18%)`	⬇️
data-pipeline-ffi	`88.36% <ø> (ø)`
ddcommon	`82.88% <ø> (ø)`
ddcommon-ffi	`69.66% <ø> (ø)`
ddtelemetry	`60.08% <ø> (ø)`
ddtelemetry-ffi	`21.32% <ø> (ø)`
dogstatsd-client	`83.26% <ø> (ø)`
datadog-ipc	`82.58% <ø> (ø)`
datadog-profiling	`77.13% <ø> (ø)`
datadog-profiling-ffi	`62.12% <ø> (ø)`
datadog-sidecar	`40.24% <ø> (ø)`
datdog-sidecar-ffi	`4.91% <ø> (ø)`
spawn-worker	`55.35% <ø> (ø)`
tinybytes	`90.96% <ø> (ø)`
datadog-trace-normalization	`98.24% <ø> (ø)`
datadog-trace-obfuscation	`94.17% <ø> (ø)`
datadog-trace-protobuf	`77.10% <ø> (ø)`
datadog-trace-utils	`89.72% <ø> (ø)`
datadog-tracer-flare	`75.14% <ø> (ø)`
datadog-log	`76.31% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ekump · 2025-07-22T20:59:29Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	70.32 MB	70.32 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	9.24 MB	9.24 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so.debug	22.12 MB	22.12 MB	0% (0 B) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	82.55 MB	82.55 MB	0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	9.16 MB	9.16 MB	0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so.debug	26.18 MB	26.18 MB	0% (0 B) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	18.40 MB	18.40 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	65.03 KB	65.03 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	124.94 MB	124.96 MB	+.01% (+16.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	646.92 MB	646.92 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	5.89 MB	5.89 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	65.03 KB	65.03 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	17.35 MB	17.35 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	32.19 MB	32.19 MB	0% (0 B) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	15.69 MB	15.69 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	66.03 KB	66.03 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	127.33 MB	127.32 MB	--.01% (-16.00 KB) 💪
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	636.65 MB	636.65 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	4.49 MB	4.49 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	66.03 KB	66.03 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	18.48 MB	18.48 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	30.23 MB	30.23 MB	0% (0 B) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	63.04 MB	63.04 MB	0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	9.85 MB	9.85 MB	0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so.debug	20.97 MB	20.97 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	77.49 MB	77.49 MB	0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	9.75 MB	9.75 MB	0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so.debug	24.10 MB	24.10 MB	0% (0 B) 👌

ekump · 2025-07-23T14:22:44Z

/merge

dd-devflow-routing-codex · 2025-07-23T14:22:49Z

View all feedbacks in Devflow UI.

2025-07-23 14:22:48 UTC ℹ️ Start processing command /merge

2025-07-23 14:23:07 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 40m (p90).

2025-07-23 15:10:20 UTC ❌ MergeQueue: The checks failed on this merge request

Tests failed on this commit 071c9b4:

cargo test --workspace #ubuntu-latest ${RUST_VERSION}

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

ekump · 2025-07-23T15:15:17Z

/merge

dd-devflow-routing-codex · 2025-07-23T15:15:21Z

View all feedbacks in Devflow UI.

2025-07-23 15:15:21 UTC ℹ️ Start processing command /merge

2025-07-23 15:15:37 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 40m (p90).

2025-07-23 15:43:43 UTC ℹ️ MergeQueue: This merge request was merged

github-actions bot added the data-pipeline label Jul 3, 2025

attempt to reduce flakiness of trace exporter shutdown tests

e5ff9fe

ekump force-pushed the ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test branch from 56b6eef to 3e8a609 Compare July 22, 2025 20:11

github-actions bot added the ci-build label Jul 22, 2025

ekump force-pushed the ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test branch from 3e8a609 to e5ff9fe Compare July 22, 2025 20:29

github-actions bot removed the ci-build label Jul 22, 2025

ekump marked this pull request as ready for review July 22, 2025 20:30

ekump requested a review from a team as a code owner July 22, 2025 20:30

VianneyRuhlmann approved these changes Jul 23, 2025

View reviewed changes

dd-devflow bot added mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: rejected and removed mergequeue-status: queued mergequeue-status: in_progress labels Jul 23, 2025

dd-devflow bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: rejected mergequeue-status: queued labels Jul 23, 2025

dd-mergequeue bot merged commit 36d16a2 into main Jul 23, 2025
37 checks passed

dd-mergequeue bot deleted the ekump/APMSP-2140-reduce-flakiness-of-test-shutdown-with-timeout-test branch July 23, 2025 15:43

dd-devflow bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Jul 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

attempt to reduce flakiness of trace exporter shutdown tests #1128

attempt to reduce flakiness of trace exporter shutdown tests #1128

Uh oh!

ekump commented Jul 3, 2025

Uh oh!

pr-commenter bot commented Jul 22, 2025 •

edited

Loading

Group 1

Group 2

Group 3

Group 4

Group 5

Group 6

Group 7

Group 8

Group 9

Group 10

Group 11

Group 12

Group 13

Uh oh!

codecov-commenter commented Jul 22, 2025

Uh oh!

ekump commented Jul 22, 2025

Uh oh!

ekump commented Jul 23, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 23, 2025 •

edited

Loading

Uh oh!

ekump commented Jul 23, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

attempt to reduce flakiness of trace exporter shutdown tests #1128

attempt to reduce flakiness of trace exporter shutdown tests #1128

Uh oh!

Conversation

ekump commented Jul 3, 2025

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

pr-commenter bot commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Candidate

Group 1

Group 2

Group 3

Group 4

Group 5

Group 6

Group 7

Group 8

Group 9

Group 10

Group 11

Group 12

Group 13

Baseline

Uh oh!

codecov-commenter commented Jul 22, 2025

Codecov Report

Uh oh!

ekump commented Jul 22, 2025

Artifact Size Benchmark Report

Uh oh!

ekump commented Jul 23, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

ekump commented Jul 23, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pr-commenter bot commented Jul 22, 2025 •

edited

Loading

dd-devflow-routing-codex bot commented Jul 23, 2025 •

edited

Loading

dd-devflow-routing-codex bot commented Jul 23, 2025 •

edited

Loading