Associate ESS metadata with job ids

[michel-laterman]

What does this PR do?

Assoicate ESS metadata with Job IDS

Why is it important?

There are timing issues inbetween FIPS and non-FIPS ESS testing where FIPS tests can retrieve data for non-FIPS deployments.

Related issues

• Closes [Flaky Test]: TestFIPS – Condition never satisfied #9033

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[michel-laterman]

I think the attempt in the earlier commit and into the same issue; i've added some echo statements so we can be sure what we're referencing.

i'll run the tests a few times to see if it re-occurs

[michel-laterman]

buildkite test this

[v1v]

IIUC, this is related to the step at

elastic-agent/.buildkite/bk.integration-fips.pipeline.yml

Lines 21 to 35 in 5ce4871

	- label: Start ESS stack for FIPS integration tests
	key: integration-fips-ess
	env:
	ASDF_TERRAFORM_VERSION: 1.9.2
	TF_VAR_integration_server_docker_image: "docker.elastic.co/beats-ci/elastic-agent-cloud-fips:git-${BUILDKITE_COMMIT:0:12}"
	command: |
	source .buildkite/scripts/steps/ess_start.sh
	artifact_paths:
	- test_infra/ess/*.tfstate
	- test_infra/ess/*.lock.hcl
	agents:
	image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-beats-ci-with-hooks:0.5"
	useCustomGlobalHooks: true
	plugins:
	- *vault_ec_key_prod

that creates a new deployment. Shall we simplify the current implementation and use the same deployment regardless?

[michel-laterman]

@v1v the deployment used by FIPS testing is different from the deployments used by normal integration tests, and custom ECH test. The difference is as follows:

• FIPS testing: deployed integrations server is built from the FIPS variant of the commit
• standard integration testing: no special instructions
• custom ECH test: deployed integrations server is built from the commit

We needed to separate FIPS from non-FIPS as the binary differs.
We wanted to separate the test with a custom integration server from normal integration tests to isolate any issues (it would be easier to debug one test if the custom server fails, then it would be to find the issue if all tests fail)

[v1v]

LGTM, however I'll wait for a second review to play safe.

I can see builds are running those new steps:

• https://buildkite.com/elastic/elastic-agent/builds/24300#01982dfc-4224-4f77-a240-414a77a19dfd/196-197
• https://buildkite.com/elastic/elastic-agent/builds/24300#01982dfc-420c-4958-bea6-29316e6a8e5a/767-768

[pkoutsovasilis]

@michel-laterman still the fips builds are not using the fips ESS stack

• fips integration testing step https://buildkite.com/elastic/elastic-agent/builds/24300#01982dfc-5b6c-4aed-bbac-204f1ae6fa95/805-807
• fips ESS provision step https://buildkite.com/elastic/elastic-agent/builds/24300#01982dfc-5b6a-439d-9308-3993dca6eae0/143-347
• non fips ESS provision step https://buildkite.com/elastic/elastic-agent/builds/24300#01982dfc-41f8-4475-8693-1a7c03753809/142-359

[v1v]

I added a couple of comments

[pkoutsovasilis]

@michel-laterman I still see mixed up non-FIPS integration test steps with FIPS stack:

• non-FIPS integration test here, and here
• non-FIPS ESS Stack provisioning here
• FIPS ESS Stack provisioning here

PS: please also print the Elasticsearch Host for tests that run on Windows

[michel-laterman]

Looks like we still have some (fips) pipeline issues:

fatal: failed to get meta-data: POST https://agent.buildkite.com/v3/jobs/019837ed-bbfe-41d6-af65-7066b4df62f8/data/get: 404 Not Found: No key "ess.job.fips" found

Looks like a test runs before the FIPS provisioning is complete

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: b99e4b3

Failed CI Steps

• fips:x86_64:sudo-false:fleet
• fips:x86_64:sudo-true:fleet
• fips:arm64:sudo-false:fleet
• fips:arm64:sudo-true:fleet

History

• 💔 Build #24344 failed e7e28d7
• 💛 Build #24319 was flaky 9523561
• 💔 Build #24300 failed b43a376
• 💔 Build #24230 failed cfb28f4
• 💔 Build #24225 failed cfb28f4

cc @michel-laterman

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube