Skip to content

Associate ESS metadata with job ids #9056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

michel-laterman
Copy link
Contributor

What does this PR do?

Assoicate ESS metadata with Job IDS

Why is it important?

There are timing issues inbetween FIPS and non-FIPS ESS testing where FIPS tests can retrieve data for non-FIPS deployments.

Related issues

@michel-laterman michel-laterman requested review from a team as code owners July 17, 2025 17:22
@michel-laterman michel-laterman added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team flaky-test Unstable or unreliable test cases. backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch labels Jul 17, 2025
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jul 17, 2025

I think the attempt in the earlier commit and into the same issue; i've added some echo statements so we can be sure what we're referencing.

i'll run the tests a few times to see if it re-occurs

@michel-laterman
Copy link
Contributor Author

buildkite test this

Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this is related to the step at

- label: Start ESS stack for FIPS integration tests
key: integration-fips-ess
env:
ASDF_TERRAFORM_VERSION: 1.9.2
TF_VAR_integration_server_docker_image: "docker.elastic.co/beats-ci/elastic-agent-cloud-fips:git-${BUILDKITE_COMMIT:0:12}"
command: |
source .buildkite/scripts/steps/ess_start.sh
artifact_paths:
- test_infra/ess/*.tfstate
- test_infra/ess/*.lock.hcl
agents:
image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-beats-ci-with-hooks:0.5"
useCustomGlobalHooks: true
plugins:
- *vault_ec_key_prod
that creates a new deployment. Shall we simplify the current implementation and use the same deployment regardless?

@michel-laterman
Copy link
Contributor Author

@v1v the deployment used by FIPS testing is different from the deployments used by normal integration tests, and custom ECH test. The difference is as follows:

  • FIPS testing: deployed integrations server is built from the FIPS variant of the commit
  • standard integration testing: no special instructions
  • custom ECH test: deployed integrations server is built from the commit

We needed to separate FIPS from non-FIPS as the binary differs.
We wanted to separate the test with a custom integration server from normal integration tests to isolate any issues (it would be easier to debug one test if the custom server fails, then it would be to find the issue if all tests fail)

v1v
v1v previously approved these changes Jul 21, 2025
Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pkoutsovasilis pkoutsovasilis self-requested a review July 22, 2025 08:42
Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a couple of comments

Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michel-laterman I still see mixed up non-FIPS integration test steps with FIPS stack:

  • non-FIPS integration test here, and here
  • non-FIPS ESS Stack provisioning here
  • FIPS ESS Stack provisioning here

PS: please also print the Elasticsearch Host for tests that run on Windows

@pkoutsovasilis pkoutsovasilis self-requested a review July 22, 2025 17:07
@michel-laterman michel-laterman requested a review from v1v July 23, 2025 20:01
@michel-laterman
Copy link
Contributor Author

Looks like we still have some (fips) pipeline issues:

fatal: failed to get meta-data: POST https://agent.buildkite.com/v3/jobs/019837ed-bbfe-41d6-af65-7066b4df62f8/data/get: 404 Not Found: No key "ess.job.fips" found

Looks like a test runs before the FIPS provisioning is complete

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 23, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

History

cc @michel-laterman

Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch flaky-test Unstable or unreliable test cases. skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Flaky Test]: TestFIPS – Condition never satisfied
4 participants