Run tests in parallel #20779
Closed
Run tests in parallel #20779
+261
−291
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
datadog-agent-integrations-bot
bot
added
dev/testing
dev/tooling
integration/postgres
ddev
labels
Codecov ReportAttention: Patch coverage is
Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds parallelization of targets with more than 4 target environments in both individual targets and the
PR Alltest.For this PR, the following optimizations have been done to avoid the Github Action maximum object size limit (see below):
standardinput. This was only false for the weekly test, for which thelatestinput can be used as a proxy for not running the full unit and integration tests.PYTHON_FILTER. This filter overrides theenvoption for running tests and forced all tests to run. It's also redundant with theSKIP_ENV_NAMEvariable, which accomplishes the same thing by filtering outpy3/2.*. Removing it also gives a small amount of head room on max object size.Note that the SQL Server on Windows test are currently broken on
masterdue to problems with the Windows runners. This is being worked on in parallel. In the meantime this PR disables those tests.Cost
There is a small amount of cost overhead in running the jobs in parallel. This should be easily offset by the time saved by being able to rerun failed tests one target environment at a time. It's also worth exploring extracting the lint job from the
test-targetinto a blocking workflow to fail faster.Github Maximum Size Details
Github Actions enforces a limit on the "maximum object size" of a test. This appears from testing to be the aggregate size of all data that goes into forming a test run. For example, splitting up tests into clusters and then calling each of those clusters in the parent
test-allfile has no impact on triggering this size.We are bumping right up against this limit for our
PR alltest. In the future we will likely need to try:test-target. Python 2 tests are infrequent and many (most?) integrations are no longer compatible with Python 2 on HEAD. Maintaining Python 2 testing support may be best handled by forking the repo, or creating a separate non-parallelized test flow.test-all-a-mandtest-all-n-zand then replace calls totest-allwith multiple jobs each calling a part. This may or may not still trigger the limit. Havingtest-allcalltest-all-0,test-all-1, etc. still triggered the max size.Motivation
Parallelizing targets with many environments massively improves turnaround time on PRs for integrations such as Postgres, which sees a typical test run go from more than an hour to ~10 minutes.
Review checklist (to be filled by reviewers)
qa/skip-qalabel if the PR doesn't need to be tested during QA.backport/<branch-name>label to the PR and it will automatically open a backport PR once this one is merged