Skip to content

ci: increase SWT-Bench default parallelism to 16 workers / 50 batch #534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
simonrosenberg wants to merge 2 commits into from
Closed

ci: increase SWT-Bench default parallelism to 16 workers / 50 batch #534

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
778a254
ci: increase SWT-Bench default parallelism to 16 workers / 50 batch
Mar 18, 2026
774d72e
ci: remove redundant env defaults, use workflow inputs as single sour…
Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 7 additions & 11 deletions .github/workflows/build-swtbench-images.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ on:
max-workers:
description: 'Maximum number of parallel workers'
required: false
default: '4'
default: '16'
type: string
build-batch-size:
description: 'Number of agent-server images to submit per batch'
required: false
default: '15'
default: '50'
type: string
n-limit:
description: 'Limit number of images to build (0 for all)'
Expand Down Expand Up @@ -91,13 +91,9 @@ jobs:
packages: write
issues: write

# Defaults for automatic runs; keep INSTANCE_IDS/SELECT_FILE initialized so set -euo pipefail won't fail on unset vars.
# Initialize vars that may be empty so set -euo pipefail won't fail on unset refs.
# All other defaults come from the workflow inputs above (single source of truth).
env:
DATASET: eth-sri/SWT-bench_Verified_bm25_27k_zsp
SPLIT: test
MAX_WORKERS: '4'
BUILD_BATCH_SIZE: '15'
N_LIMIT: '0'
INSTANCE_IDS: ''
SELECT_FILE: ''

Expand Down Expand Up @@ -155,8 +151,8 @@ jobs:
# Get inputs with defaults
DATASET="${{ inputs.dataset || 'eth-sri/SWT-bench_Verified_bm25_27k_zsp' }}"
SPLIT="${{ inputs.split || 'test' }}"
MAX_WORKERS="${{ inputs.max-workers || '4' }}"
BUILD_BATCH_SIZE="${{ inputs.build-batch-size || '15' }}"
MAX_WORKERS="${{ inputs.max-workers || '16' }}"
BUILD_BATCH_SIZE="${{ inputs.build-batch-size || '50' }}"
N_LIMIT="${{ inputs.n-limit || '0' }}"
INSTANCE_IDS="${{ inputs.instance-ids }}"
FORCE_BUILD="${{ inputs.force-build || 'false' }}"
Expand Down Expand Up @@ -218,7 +214,7 @@ jobs:
N_LIMIT="${{ inputs.n-limit || '0' }}"
INSTANCE_IDS="${{ inputs.instance-ids }}"
IMAGE_PREFIX="${{ inputs.eval-image-prefix || 'ghcr.io/openhands/swtbench-eval' }}"
MAX_WORKERS="${{ inputs.max-workers || '4' }}"
MAX_WORKERS="${{ inputs.max-workers || '16' }}"
BUILD_MODE="${{ inputs.build-mode || 'cli' }}"
MAX_RETRIES="${{ inputs.max-retries || '2' }}"
BUILD_BATCH_SIZE="${{ inputs.eval-env-build-batch-size || '10' }}"
Expand Down
Loading