Skip to content

Prefix BK metadata with fips indicator #9056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 25, 2025

Conversation

michel-laterman
Copy link
Contributor

@michel-laterman michel-laterman commented Jul 17, 2025

What does this PR do?

Prefix BK metadata with fips indicator

Why is it important?

There are timing issues inbetween FIPS and non-FIPS ESS testing where FIPS tests can retrieve data for non-FIPS deployments.

Related issues

@michel-laterman michel-laterman requested review from a team as code owners July 17, 2025 17:22
@michel-laterman michel-laterman added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team flaky-test Unstable or unreliable test cases. backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch labels Jul 17, 2025
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Jul 17, 2025

I think the attempt in the earlier commit and into the same issue; i've added some echo statements so we can be sure what we're referencing.

i'll run the tests a few times to see if it re-occurs

@michel-laterman
Copy link
Contributor Author

buildkite test this

Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this is related to the step at

- label: Start ESS stack for FIPS integration tests
key: integration-fips-ess
env:
ASDF_TERRAFORM_VERSION: 1.9.2
TF_VAR_integration_server_docker_image: "docker.elastic.co/beats-ci/elastic-agent-cloud-fips:git-${BUILDKITE_COMMIT:0:12}"
command: |
source .buildkite/scripts/steps/ess_start.sh
artifact_paths:
- test_infra/ess/*.tfstate
- test_infra/ess/*.lock.hcl
agents:
image: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-beats-ci-with-hooks:0.5"
useCustomGlobalHooks: true
plugins:
- *vault_ec_key_prod
that creates a new deployment. Shall we simplify the current implementation and use the same deployment regardless?

@michel-laterman
Copy link
Contributor Author

@v1v the deployment used by FIPS testing is different from the deployments used by normal integration tests, and custom ECH test. The difference is as follows:

  • FIPS testing: deployed integrations server is built from the FIPS variant of the commit
  • standard integration testing: no special instructions
  • custom ECH test: deployed integrations server is built from the commit

We needed to separate FIPS from non-FIPS as the binary differs.
We wanted to separate the test with a custom integration server from normal integration tests to isolate any issues (it would be easier to debug one test if the custom server fails, then it would be to find the issue if all tests fail)

v1v
v1v previously approved these changes Jul 21, 2025
Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pkoutsovasilis pkoutsovasilis self-requested a review July 22, 2025 08:42
Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a couple of comments

Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michel-laterman I still see mixed up non-FIPS integration test steps with FIPS stack:

  • non-FIPS integration test here, and here
  • non-FIPS ESS Stack provisioning here
  • FIPS ESS Stack provisioning here

PS: please also print the Elasticsearch Host for tests that run on Windows

@pkoutsovasilis pkoutsovasilis self-requested a review July 22, 2025 17:07
@michel-laterman michel-laterman requested a review from v1v July 23, 2025 20:01
@michel-laterman
Copy link
Contributor Author

Looks like we still have some (fips) pipeline issues:

fatal: failed to get meta-data: POST https://agent.buildkite.com/v3/jobs/019837ed-bbfe-41d6-af65-7066b4df62f8/data/get: 404 Not Found: No key "ess.job.fips" found

Looks like a test runs before the FIPS provisioning is complete

Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

proposed a new approach, lemme know if it makes sense to you @michel-laterman

@pkoutsovasilis pkoutsovasilis self-requested a review July 24, 2025 07:27
Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michel-laterman please add

FIPS: "true"

here

steps:
- label: Start ESS stack for FIPS integration tests
key: integration-fips-ess
env:
ASDF_TERRAFORM_VERSION: 1.9.2
TF_VAR_integration_server_docker_image: "docker.elastic.co/beats-ci/elastic-agent-cloud-fips:git-${BUILDKITE_COMMIT:0:12}"
command: |
source .buildkite/scripts/steps/ess_start.sh
artifact_paths:
- test_infra/ess/*.tfstate

@pkoutsovasilis pkoutsovasilis self-requested a review July 24, 2025 18:33
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

History

cc @michel-laterman

Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@michel-laterman michel-laterman changed the title Associate ESS metadata with job ids Prefix BK metadata with fips indicator Jul 25, 2025
@michel-laterman michel-laterman merged commit 4b01c0e into elastic:main Jul 25, 2025
19 checks passed
@michel-laterman michel-laterman deleted the ess-job-id branch July 25, 2025 13:16
mergify bot pushed a commit that referenced this pull request Jul 25, 2025
Prefix BK metadata with fips indicator

Co-authored-by: Panos Koutsovasilis <[email protected]>
(cherry picked from commit 4b01c0e)
mergify bot pushed a commit that referenced this pull request Jul 25, 2025
Prefix BK metadata with fips indicator

Co-authored-by: Panos Koutsovasilis <[email protected]>
(cherry picked from commit 4b01c0e)
michel-laterman added a commit that referenced this pull request Jul 25, 2025
Prefix BK metadata with fips indicator


(cherry picked from commit 4b01c0e)

Co-authored-by: Michel Laterman <[email protected]>
Co-authored-by: Panos Koutsovasilis <[email protected]>
michel-laterman added a commit that referenced this pull request Jul 25, 2025
Prefix BK metadata with fips indicator


(cherry picked from commit 4b01c0e)

Co-authored-by: Michel Laterman <[email protected]>
Co-authored-by: Panos Koutsovasilis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch flaky-test Unstable or unreliable test cases. skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Flaky Test]: TestFIPS – Condition never satisfied
4 participants