Add official system-tests workflow on pushes on master

[cbeauchesne]

What does this PR do?

Use the official system-tests WF for pushes on master branch to run end-to-end and parametric scenarios, to allow a comparizon with the current setup

Motivation

dd-trace-rb use a custom workflow which is hard to maintain

Comparizon

item	Current	Proposal
weblog X scenario count	122	approx 158
CI time run 1	9'24 (with a docker rate limit error)	9'20
CI time run 2	9'16	10'05
CI time run 3	9'35	8'49

Notes

• this new workflow only runs on pushes on master
• The capacity to define a very customized set of weblog X scenario is more limited. I used the closest set that contains all existing one in the current workflow, with a reasonnable amount of definition, that's why there are 158 runs, vs 122
  • long term, if the guild choose the official WF, I strongly recommend to use the tracer-release groups, rather hardcoding scenario names
• if we run all 1600 set of weblog X scenario (so 15 times more than actually), it takes 20mn (tried several times in this PR, no issue on docker rate limit).

Change log entry
None.

Additional Notes:

How to test the change?

[github-actions]

Thank you for updating Change log entry section 👏

^{Visited at: 2025-07-02 10:07:54 UTC}

[ivoanjo]

To add context to this PR: this is an experiment so we can do an apples-to-apples comparison with the existing integration.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.52%. Comparing base (1a043b0) to head (5a8cc4f).
Report is 41 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4774      +/-   ##
==========================================
- Coverage   97.54%   97.52%   -0.02%     
==========================================
  Files        1484     1483       -1     
  Lines       88499    88601     +102     
  Branches     4588     4592       +4     
==========================================
+ Hits        86322    86412      +90     
- Misses       2177     2189      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-07-08 15:00:51

Comparing candidate commit 5a8cc4f in PR branch cbeauchesne/system-tests-official with baseline commit 1a043b0 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 6 unstable metrics.

[p-datadog]

Two questions:

1. Why does this workflow not run on pull requests, if we are attempting an "apples to apples" comparison?

2. Does the exsiting runner run "1600 weblog X scenario"?

[p-datadog]

And one more question: does this PR authenticate to docker hub, if so how, if not how is it not subject to rate limits?

[cbeauchesne]

And one more question: does this PR authenticate to docker hub, if so how, if not how is it not subject to rate limits?

It does, the workflow inherits from your secrets, and if docker username/password are present, it do the authentification : https://github.com/DataDog/system-tests/blob/main/.github/workflows/run-end-to-end.yml#L107-L115

[cbeauchesne]

Why does this workflow not run on pull requests, if we are attempting an "apples to apples" comparison?

To prevent any issue on PRs, and being able to decide to use it on PR after few days of runs on master.

Does the exsiting runner run "1600 weblog X scenario"?

No, it run only 122 of them (details in PR description)

[ivoanjo]

Oh, I must admit it was quite surprising to me that setting up this flow is ~130 lines of code -- for CI stuff I'm always fearing a bit of too much verbosity.

I don't see any issue, but I'll hold on pressing the approve button as this is a big change, so I think it makes sense to have a bit more time for the guild folks to chime in and put this through its paces.

Also, doesn't have to be in this PR, but I think once we agree to merge this, it makes sense to also remove the old integration (or at least disable it) so we don't re-run both sets of tests via both integrations.

[vpellan]

Is it possible to add support for the forced-tests-list.json file ? I use it often, and at least @Strech and @marcotc have also used it. I can help if that's needed, I also did something similar for parametric-tests.yml, which uses system-tests official workflow

[p-datadog]

Where would I find in the logs the confirmation that docker hub requests are authenticated?

[cbeauchesne]

Where would I find in the logs the confirmation that docker hub requests are authenticated?

https://github.com/DataDog/dd-trace-rb/actions/runs/16024726999/job/45210078369#step:8:69

[cbeauchesne]

Is it possible to add support for the forced-tests-list.json file ? I use it often, and at least @Strech and @marcotc have also used it. I can help if that's needed, I also did something similar for parametric-tests.yml, which uses system-tests official workflow

@vpellan Yes please :) ping me if you need any help

[p-datadog]

I was under the impression that this PR is the complete work, but if it's a partial change I have no objections.

Currently outstanding work items as I see them for the testing vs ST workflow to start:

1. ST workflow needs to run on pull requests
2. ST workflow needs to run all tests (including all of the ones presently being run by our workflow)
3. ST workflow needs to support forced tests as mentioned by @vpellan

[ivoanjo]

I'm curious about:

2. ST workflow needs to cover all of the configurations that our workflow covers

I thought it did? (I might be misunderstanding some part)

[p-datadog]

I did not say the right thing and updated the comment: this PR claims to run all tests that our present workflow runs but the idea for changing to ST workflow was that it would run all tests so that we cannot have unintentionally un-executed tests.

So the ST workflow should not require an enumeration of tests to run as this PR is presently doing.

[ivoanjo]

[...] the idea for changing to ST workflow was that it would run all tests so that we cannot have unintentionally un-executed tests.

So the ST workflow should not require an enumeration of tests to run as this PR is presently doing.

Got it, thanks for explaining. TBH I was the one that pushed for a more apples-to-apples comparison, as in our discussions so far we had a lot of (very fair) concerns about CI performance regressions with the changeover.

So that one's on me! As you see in the current PR, it's more work to match our current set than it would be to run them all.

That said, enabling all tests is definitely a trade-off, and does come with a clear CI time regression (+10 minutes). So my suggestion would be:

1. Put that constraint aside for this PR
2. Use to use the next Ruby guild meeting to decide if we're happy with the trade-off here

(To be clear: I really like speedy CI, but am happy to take the increased test coverage instead)

If we decide we're happy, it seems trivial to have a follow-up to enable all the tests.

[cbeauchesne]

There is also another point : @lloeki told on slack that it's useless to run the 1600 matrix point.

As it's also a waste of ressource inside system-tests CI, I will define the set of useful matrix point inside system-tests.

It means that if the guild chose to go with this official workflow, the definition will be way more simple - and still fast.