ci: increase SWT-Bench default parallelism to 16 workers / 50 batch#534
Closed
simonrosenberg wants to merge 2 commits intomainfrom
Closed
ci: increase SWT-Bench default parallelism to 16 workers / 50 batch#534simonrosenberg wants to merge 2 commits intomainfrom
simonrosenberg wants to merge 2 commits intomainfrom
Conversation
Raise the default `max-workers` from 4 to 16 and `build-batch-size` from 15 to 50 for the SWT-Bench image build workflow. The previous defaults were reduced as a safety measure during the image build regression investigation. With the 24-hour timeout (#528) and sdist caching (#515) now in place, higher parallelism can be restored. The inputs remain fully configurable for manual runs that need different settings. Related: #531 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
all-hands-bot
approved these changes
Mar 18, 2026
Collaborator
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
🟢 Good taste - This is fine. You're changing numbers in a YAML file after fixing the root causes. The defaults are duplicated across inputs/env/fallbacks, but that's how GitHub Actions workflow_dispatch works — over-engineering a "single source of truth" would add more complexity than the duplication costs. Easily reversible if the parallelism causes issues. Ship it.
…ce of truth The job-level env block duplicated default values already defined in the workflow inputs. Every step that uses these vars re-assigns them from inputs with inline fallbacks, so the env defaults were never actually read. Remove DATASET, SPLIT, MAX_WORKERS, BUILD_BATCH_SIZE, and N_LIMIT from the env block. Keep only INSTANCE_IDS and SELECT_FILE which need initialization for set -euo pipefail (they can be legitimately empty). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Closing — the defaults are already configurable via workflow_dispatch inputs. Before changing defaults, we should do a proper grid search to find the optimal values. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Raises the default parallelism for the SWT-Bench image build workflow:
max-workers: 4 → 16build-batch-size: 15 → 50These were reduced to 4/15 as a safety measure during the image build regression investigation. With the 24-hour timeout (#528) and sdist caching (#515) now merged, higher parallelism can be restored.
The inputs remain fully configurable — manual
workflow_dispatchruns can still override to any value (e.g.,max-workers=4for debugging).What this PR does NOT change:
OPENHANDS_BUILDKIT_CACHE_MODEis not set — SDK defaults apply)Test plan
Related: #531