Associate ESS metadata with job ids #9056

michel-laterman · 2025-07-17T17:22:03Z

What does this PR do?

Assoicate ESS metadata with Job IDS

Why is it important?

There are timing issues inbetween FIPS and non-FIPS ESS testing where FIPS tests can retrieve data for non-FIPS deployments.

Related issues

Closes [Flaky Test]: TestFIPS – Condition never satisfied #9033

elasticmachine · 2025-07-17T17:22:07Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

michel-laterman · 2025-07-17T19:05:02Z

I think the attempt in the earlier commit and into the same issue; i've added some echo statements so we can be sure what we're referencing.

i'll run the tests a few times to see if it re-occurs

michel-laterman · 2025-07-17T21:12:49Z

buildkite test this

.buildkite/scripts/steps/integration_tests_tf.sh

v1v

IIUC, this is related to the step at

elastic-agent/.buildkite/bk.integration-fips.pipeline.yml

Lines 21 to 35 in 5ce4871

    
             - label: Start ESS stack for FIPS integration tests 
        
               key: integration-fips-ess 
        
               env: 
        
                 ASDF_TERRAFORM_VERSION: 1.9.2 
        
                 TF_VAR_integration_server_docker_image: "docker.elastic.co/beats-ci/elastic-agent-cloud-fips:git-${BUILDKITE_COMMIT:0:12}" 
        
               command: | 
        
                 source .buildkite/scripts/steps/ess_start.sh 
        
               artifact_paths: 
        
                 - test_infra/ess/*.tfstate 
        
                 - test_infra/ess/*.lock.hcl 
        
               agents: 
        
                 image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-beats-ci-with-hooks:0.5" 
        
                 useCustomGlobalHooks: true 
        
               plugins: 
        
                 - *vault_ec_key_prod

that creates a new deployment. Shall we simplify the current implementation and use the same deployment regardless?

.buildkite/scripts/steps/integration_tests_tf.sh

michel-laterman · 2025-07-18T14:09:59Z

@v1v the deployment used by FIPS testing is different from the deployments used by normal integration tests, and custom ECH test. The difference is as follows:

FIPS testing: deployed integrations server is built from the FIPS variant of the commit
standard integration testing: no special instructions
custom ECH test: deployed integrations server is built from the commit

We needed to separate FIPS from non-FIPS as the binary differs.
We wanted to separate the test with a custom integration server from normal integration tests to isolate any issues (it would be easier to debug one test if the custom server fails, then it would be to find the issue if all tests fail)

.buildkite/scripts/steps/ess.ps1

v1v

LGTM, however I'll wait for a second review to play safe.

I can see builds are running those new steps:

pkoutsovasilis

@michel-laterman still the fips builds are not using the fips ESS stack

fips integration testing step https://buildkite.com/elastic/elastic-agent/builds/24300#01982dfc-5b6c-4aed-bbac-204f1ae6fa95/805-807
fips ESS provision step https://buildkite.com/elastic/elastic-agent/builds/24300#01982dfc-5b6a-439d-9308-3993dca6eae0/143-347
non fips ESS provision step https://buildkite.com/elastic/elastic-agent/builds/24300#01982dfc-41f8-4475-8693-1a7c03753809/142-359

v1v

I added a couple of comments

.buildkite/scripts/steps/ess.ps1

.buildkite/scripts/steps/ess_start.sh

.buildkite/scripts/steps/integration_tests_tf.sh

.buildkite/scripts/steps/ess_start.sh

pkoutsovasilis

@michel-laterman I still see mixed up non-FIPS integration test steps with FIPS stack:

non-FIPS integration test here, and here
non-FIPS ESS Stack provisioning here
FIPS ESS Stack provisioning here

PS: please also print the Elasticsearch Host for tests that run on Windows

michel-laterman · 2025-07-23T20:04:27Z

Looks like we still have some (fips) pipeline issues:

fatal: failed to get meta-data: POST https://agent.buildkite.com/v3/jobs/019837ed-bbfe-41d6-af65-7066b4df62f8/data/get: 404 Not Found: No key "ess.job.fips" found

Looks like a test runs before the FIPS provisioning is complete

pkoutsovasilis

proposed a new approach, lemme know if it makes sense to you @michel-laterman

.buildkite/scripts/steps/ess_start.sh

.buildkite/scripts/steps/integration_tests_tf.sh

.buildkite/scripts/steps/ess.ps1

Co-authored-by: Panos Koutsovasilis <[email protected]>

pkoutsovasilis

@michel-laterman please add

FIPS: "true"

here

elastic-agent/.buildkite/bk.integration-fips.pipeline.yml

Lines 20 to 29 in 7b95178

    
           steps: 
        
             - label: Start ESS stack for FIPS integration tests 
        
               key: integration-fips-ess 
        
               env: 
        
                 ASDF_TERRAFORM_VERSION: 1.9.2 
        
                 TF_VAR_integration_server_docker_image: "docker.elastic.co/beats-ci/elastic-agent-cloud-fips:git-${BUILDKITE_COMMIT:0:12}" 
        
               command: | 
        
                 source .buildkite/scripts/steps/ess_start.sh 
        
               artifact_paths: 
        
                 - test_infra/ess/*.tfstate

elastic-sonarqube · 2025-07-24T19:57:51Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

elasticmachine · 2025-07-24T20:16:52Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 07d92a9

Failed CI Steps

x86_64:non-sudo: default

History

💛 Build #24391 was flaky 50a28ca
💔 Build #24390 failed 3005695
💛 Build #24360 was flaky b99e4b3
💔 Build #24344 failed e7e28d7
💛 Build #24319 was flaky 9523561
💔 Build #24300 failed b43a376

cc @michel-laterman

pkoutsovasilis

LGTM

v1v

Thanks

Associate ESS metadata with job ids

556329a

michel-laterman requested a review from pkoutsovasilis July 17, 2025 17:22

michel-laterman requested review from a team as code owners July 17, 2025 17:22

michel-laterman added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team flaky-test Unstable or unreliable test cases. backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch labels Jul 17, 2025

mergify bot assigned michel-laterman Jul 17, 2025

Echo job ids

cfb28f4

michel-laterman added the skip-changelog label Jul 17, 2025

pkoutsovasilis reviewed Jul 17, 2025

View reviewed changes

.buildkite/scripts/steps/integration_tests_tf.sh Outdated Show resolved Hide resolved

v1v reviewed Jul 18, 2025

View reviewed changes

.buildkite/scripts/steps/integration_tests_tf.sh Outdated Show resolved Hide resolved

Add --job in powershell scripts

b43a376

michel-laterman commented Jul 21, 2025

View reviewed changes

.buildkite/scripts/steps/ess.ps1 Outdated Show resolved Hide resolved

v1v previously approved these changes Jul 21, 2025

View reviewed changes

pkoutsovasilis reviewed Jul 22, 2025

View reviewed changes

pkoutsovasilis self-requested a review July 22, 2025 08:42

Correct JOB_ID assignment

9523561

michel-laterman dismissed v1v’s stale review via 9523561 July 22, 2025 14:39

v1v reviewed Jul 22, 2025

View reviewed changes

pkoutsovasilis reviewed Jul 22, 2025

View reviewed changes

pkoutsovasilis self-requested a review July 22, 2025 17:07

Review feedback

e7e28d7

michel-laterman requested a review from v1v July 23, 2025 20:01

use plugin for fips deployment upgrade test

b99e4b3

pkoutsovasilis reviewed Jul 24, 2025

View reviewed changes

.buildkite/scripts/steps/ess_start.sh Outdated Show resolved Hide resolved

.buildkite/scripts/steps/integration_tests_tf.sh Outdated Show resolved Hide resolved

.buildkite/scripts/steps/ess.ps1 Outdated Show resolved Hide resolved

pkoutsovasilis self-requested a review July 24, 2025 07:27

michel-laterman and others added 3 commits July 24, 2025 07:36

Apply suggestions from code review

2c99659

Co-authored-by: Panos Koutsovasilis <[email protected]>

Cleanup ess.ps1

3005695

Fix script

50a28ca

pkoutsovasilis reviewed Jul 24, 2025

View reviewed changes

Add FIPS: true to bk fips stack provisioning

07d92a9

pkoutsovasilis self-requested a review July 24, 2025 18:33

pkoutsovasilis approved these changes Jul 24, 2025

View reviewed changes

v1v approved these changes Jul 25, 2025

View reviewed changes

	- label: Start ESS stack for FIPS integration tests
	key: integration-fips-ess
	env:
	ASDF_TERRAFORM_VERSION: 1.9.2
	TF_VAR_integration_server_docker_image: "docker.elastic.co/beats-ci/elastic-agent-cloud-fips:git-${BUILDKITE_COMMIT:0:12}"
	command: \|
	source .buildkite/scripts/steps/ess_start.sh
	artifact_paths:
	- test_infra/ess/*.tfstate
	- test_infra/ess/*.lock.hcl
	agents:
	image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-beats-ci-with-hooks:0.5"
	useCustomGlobalHooks: true
	plugins:
	- *vault_ec_key_prod

	steps:
	- label: Start ESS stack for FIPS integration tests
	key: integration-fips-ess
	env:
	ASDF_TERRAFORM_VERSION: 1.9.2
	TF_VAR_integration_server_docker_image: "docker.elastic.co/beats-ci/elastic-agent-cloud-fips:git-${BUILDKITE_COMMIT:0:12}"
	command: \|
	source .buildkite/scripts/steps/ess_start.sh
	artifact_paths:
	- test_infra/ess/*.tfstate

Associate ESS metadata with job ids #9056

Are you sure you want to change the base?

Associate ESS metadata with job ids #9056

Conversation

michel-laterman commented Jul 17, 2025

What does this PR do?

Why is it important?

Related issues

Uh oh!

elasticmachine commented Jul 17, 2025

Uh oh!

michel-laterman commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michel-laterman commented Jul 17, 2025

Uh oh!

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michel-laterman commented Jul 18, 2025

Uh oh!

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

pkoutsovasilis left a comment

Choose a reason for hiding this comment

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pkoutsovasilis left a comment

Choose a reason for hiding this comment

Uh oh!

michel-laterman commented Jul 23, 2025

Uh oh!

pkoutsovasilis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pkoutsovasilis left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Jul 24, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Jul 24, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

History

Uh oh!

pkoutsovasilis left a comment

Choose a reason for hiding this comment

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michel-laterman commented Jul 17, 2025 •

edited

Loading