Revert commit 2bfcc6c to speed up image builds

[juanmichelini]

Images are building slow and timeing out, testing shows the slowdown appears with commit 2bfcc6c
We have tried to keep the refactor without slowing down not creating other issues without luck.

For reference #486,
#487, #499

Commit 2bfcc6c is a good refactor, but we need to restore the images builds to run full benchmarks.

[juanmichelini]

@OpenHands Let's create a PR that cleanly reverts 2bfcc6c

We only want that change nothing else, no tests, no nothing except a revert. Mark it as draft and I'll test it manually.

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary

I have successfully completed the task of creating a clean revert of commit 2bfcc6c.

What was done:

✅ Created a clean revert of commit 2bfcc6c
✅ No additional changes - only the revert itself (no new tests, no other modifications)
✅ Marked as draft PR for manual testing as requested
✅ Already pushed to branch revert-2bfcc6c-image-build-speedup

Draft PR Created:

#503

Files affected (12 total):

• 7 benchmark run_infer.py files (commit0, gaia, swtbench, swebench, multiswebench, swebenchmultimodal, swefficiency)
• 2 utility files (build_utils.py, image_utils.py)
• 1 test file removed (test_image_utils.py - this was added by the original commit)
• 2 other files (test_llm_config.py, vendor/software-agent-sdk submodule)

The PR is ready for your manual testing to verify that reverting the auto-detect Docker images feature resolves the image build timeout issues mentioned in issue #502.

[simonrosenberg]

Deep-Dive Analysis: Image Build Slowdowns from Commit 2bfcc6c

I've done a thorough investigation of the codebase, GitHub Actions history, build logs, and the chain of attempted fixes. Here's a consolidated analysis and actionable next steps.

---

Root Cause: Not One Bug, But Three Compounding Issues

Commit 2bfcc6c (PR #456) introduced three independent problems that compound when building 500 images:

1. BuildKit Cache Invalidation (the biggest factor)

PR #456 updated the SDK submodule to a new commit (bde715c1). This has two cascading effects:

• Image tag mismatch: Image tags include the SDK SHA (e.g., 8ac732e-sweb.eval.x86_64.django...). A new SDK SHA means remote_image_exists() returns false for all 500 images, forcing full rebuilds even if the actual image contents are identical.
• buildcache-* invalidation: The SDK's Docker build uses per-image registry cache tags (cache-from/cache-to). When the SDK's Dockerfile dependency graph changes (e.g., uv sync --extra boto3 was added), these caches become cold.

Evidence (from @neubig's excellent analysis in #476):

• Warm-cache full 500-image builds: 55 GiB disk, ~50 min (Jan 8, Feb 19-20)
• Cold-cache full 500-image builds: 594+ GiB disk, 5+ hours (Feb 5, Mar 3-4)
• The difference is NOT whether images are skipped -- both categories had true 500-build runs with 0 skips. The difference is BuildKit layer cache effectiveness.

Current impact today: I checked the 3 currently in-progress builds (runs 22963995490, 22967520137, 22971348588). All have been stuck in "Build and push SWE-Bench images" for 1.5-4.5+ hours. They each use a different SDK SHA, which means they're all cold-cache rebuilds. Meanwhile, today's "successful" builds (22953401384, 22944417028) only built 5-10 images each because images already existed for those specific SDK SHAs.

2. uv build --sdist Per-Image Overhead (~2 hours)

As identified in PR #499: each image rebuild runs uv build --sdist for the SDK, taking ~15 seconds per image. For a full 500-image cold-cache build, this adds ~2 hours of pure CPU/IO overhead just for packaging -- before any Docker build even starts. This overhead didn't exist before 2bfcc6c because the previous SDK submodule version didn't trigger it at this scale.

3. Fork + Threading Deadlock Risk

As identified in PR #486: the new from openhands.sdk import get_logger in image_utils.py initializes RichHandler with locks/threads at import time. When build_all_images() uses ProcessPoolExecutor with fork() (Linux default), child processes can inherit locked mutexes, causing deadlock. This is the same class of bug that PR #459 fixed in evaluation.py, but the fix was never applied to build_utils.py.

@neubig's analysis suggests the March builds were "extreme slowdown with long prune stalls" rather than hard deadlocks (manifest entries show progress was being made). However, the deadlock risk is real and may explain some of the worst-case hangs.

---

Why PRs #486, #487, #499 Didn't Fully Solve It

PR	Approach	Issue
#486	Use spawn multiprocessing	Fixes deadlock but OOM concerns -- spawn creates separate memory spaces per worker
#487	Disable rich logging (LOG_JSON=1)	Fixes deadlock but introduced 75k+ WARNING lines per image (SDK logs every file copy from uv build). Later commits tried to suppress these, but log suppression doesn't fix the 15s/image uv build overhead
#499	Pre-build SDK sdist once, monkeypatch to reuse	Addresses the uv build overhead. Most promising. Latest commit stores sdist in build_dir instead of /tmp to avoid OOM. But still not merged/tested at 500-image scale

Each PR addresses a different facet but none addresses all three simultaneously. The cache invalidation issue (factor #1) is the biggest contributor and none of the PRs address it.

---

Recommendation: Staged Approach

Short-term (Unblock Now)

Merge PR #503 (revert) to restore builds to their known-good state. This is the right call given that builds are actively stuck. The revert:

• Restores the old SDK submodule SHA → images with existing tags will be cache-warm again
• Removes the get_logger import → eliminates deadlock risk
• Removes the auto-detect pattern → restores original build flow

Medium-term (Re-introduce the Refactoring Safely)

When re-introducing 2bfcc6c's improvements, address all three issues:

1. Decouple SDK submodule updates from the refactoring PR. The auto-detect pattern (create_docker_workspace, ensure_local_image) is a code-only change. The SDK submodule bump should be a separate PR so cache invalidation is a deliberate, expected event.

2. Adopt the sdist-reuse pattern from PR fix: Reuse pre-built SDK sdist to speed up image builds #499 before doing any large-scale SDK submodule bump. Pre-building the SDK sdist once eliminates ~2 hours of overhead. This should be merged independently, validated at 500-image scale, and become the baseline before changing SDK versions.

3. Fix fork safety in one of two ways:

   • Option A (simplest): Lazy-import get_logger inside functions rather than at module level in image_utils.py. This avoids RichHandler initialization before fork.
   • Option B: Use LOG_JSON=1 in the workflow env (no sitecustomize.py needed) + suppress SDK build logger at ERROR level. PR Disable rich logging by default to fix multiprocessing deadlock #487's approach but simplified.
   • Option C: Switch to spawn in build_utils.py, but with max_workers tuned to avoid OOM (e.g., reduce from 12 to 6-8).

4. Add a workflow-level timeout to the "Build and push" step (e.g., timeout-minutes: 180). Today's stuck builds have been running for 4+ hours with no automatic cancellation. A timeout would at least surface failures quickly instead of silently consuming runner time.

Long-term (Prevent Recurrence)

5. Cache-warming strategy. When the SDK submodule is intentionally bumped, run a small seed build first (e.g., 15-30 images) to warm the buildcache-* tags before dispatching the full 500-image build. This turns a cold-cache run into a warm-cache run for the subsequent full build.

6. Consider making image tags SDK-SHA-independent for cases where only the SDK code changed but the base image content is identical. The current tagging scheme ({sdk_sha}-{base_image}-{target}) forces a full rebuild every time the SDK SHA changes, even if the Dockerfile graph hasn't changed in a way that matters.

---

Experiments to Validate

Before re-introducing the refactoring, these experiments would build confidence:

1. Baseline: After merging Revert "feat: auto-detect pre-built Docker images across all benchmarks (#456)" #503, dispatch a full 500-image build with the default SDK submodule. Verify it completes in < 60 min.

2. sdist-reuse validation: Apply PR fix: Reuse pre-built SDK sdist to speed up image builds #499's sdist-reuse patch on top of the reverted code, then dispatch a full 500-image build with a new SDK SHA. Measure the sdist overhead reduction.

3. Fork safety validation: On a test branch, re-introduce only the get_logger import (without SDK bump), and dispatch a 500-image build with LOG_JSON=1 in the workflow env. Verify no deadlock.

4. Combined validation: Only after experiments 2 and 3 succeed independently, combine them and dispatch a full 500-image cold-cache build. This is the true test.

[simonrosenberg]

Correction: Deeper Investigation — The SDK Submodule Bump Is the Primary Suspect

After more critical analysis, I believe my earlier comment accepted several unvalidated theories. Here's what I found when digging deeper.

---

The SDK submodule bump is much larger than previously discussed

Commit 2bfcc6c bumped the SDK submodule from b498a699 → bde715c1. This spans 176 commits of SDK changes. The critical change is in the SDK's Dockerfile:

-ARG BASE_IMAGE=nikolaik/python-nodejs:python3.13-nodejs22
+ARG BASE_IMAGE=nikolaik/python-nodejs:python3.12-nodejs22

-FROM python:3.13-bullseye AS builder
+FROM python:3.12-bullseye AS builder

-    uv python install 3.13 && uv venv --python 3.13 .venv && uv sync --frozen --no-editable --managed-python
+    uv python install 3.12 && uv venv --python 3.12 .venv && uv sync --frozen --no-editable --managed-python --extra boto3

This is a fundamental change to every Docker layer: different Python version (3.13→3.12), different base images at every stage, and --extra boto3 adding significant new dependencies to uv sync. This would invalidate 100% of BuildKit layer cache for all 500 images — not just the image-level skip cache, but the layer-level cache that --cache-from type=registry relies on.

The uv build --sdist overhead theory (PR #499) needs re-examination

PR #499 claims uv build --sdist takes ~15 seconds per image and is a new bottleneck. But both the old and new SDK versions have uv build --sdist in the same code path (_make_build_context() → uv build --sdist). It existed before 2bfcc6c. So the question isn't "why is sdist being called" — it's "why does sdist (or everything around it) take longer now?"

Possible explanations:

• The new SDK with --extra boto3 may produce a larger sdist tarball, making packaging and extraction slower
• The Python 3.12 base may behave differently with uv build
• But 15s × 500 images ÷ 12 workers ≈ 10 minutes — significant but nowhere near the 5-hour slowdown. The sdist overhead alone doesn't explain the problem.

The fork+deadlock theory (PR #486) isn't supported by evidence

The from openhands.sdk import get_logger import in image_utils.py IS new (added by 2bfcc6c) and DOES trigger RichHandler initialization before ProcessPoolExecutor forks workers. This is a real code concern.

However, @neubig's analysis of the actual March build logs shows:

• Run 22627544120 produced 469/500 manifest entries before cancellation
• Run 22694639473 produced 291/500 entries before cancellation
• Workers were making continuous progress, not stuck in mutex waits

This is inconsistent with a hard deadlock. The builds are slow, not deadlocked. The get_logger import may cause intermittent contention, but it's not the primary cause of the 10x slowdown.

The real question: Why were cold-cache builds fast before but slow now?

This is the critical unsolved question. From @neubig's data:

Date	SDK	Builds/Skips	BuildKit Disk	Time	Result
Jan 8	old	500 / 0	55 GiB	52 min	success
Feb 20	old	500 / 0	57 GiB	2h47m	success
Mar 3	new (217b218)	469 / 15	594 GiB	5h56m	cancelled
Mar 4	new (5015e72)	291 / 0	599 GiB	5h60m	cancelled

Both Jan 8 and Mar 3 are cold-cache, full 500-image rebuilds (0 or near-0 remote skips). But Jan 8 used 55 GiB and finished in 52 min, while Mar 3 used 594 GiB (10x more) and took 5h56m (7x slower).

The 10x disk footprint increase points directly at the Dockerfile dependency changes. With --extra boto3 and Python 3.12, each image build now materializes significantly more data in the BuildKit cache. At 500 images, this pushes disk usage past the prune threshold, triggering slow docker buildx prune operations that dominate total build time.

Summary of what's actually happening

The primary cause is the SDK Dockerfile changes (Python 3.13→3.12 + --extra boto3) which:

1. Invalidate all existing BuildKit layer caches (100% cache miss)
2. Produce ~10x larger per-image disk footprint in BuildKit
3. Push total disk usage past the prune threshold (60 GiB target / 60% trigger)
4. Cause frequent, slow docker buildx prune operations between batches
5. Each prune stall multiplies across 33+ batches (500 images ÷ 15/batch)

The get_logger import adds a secondary risk of fork contention but is NOT the primary bottleneck.

The uv build --sdist runs per-image in both old and new SDK versions. It may be marginally slower with the larger SDK but doesn't explain the 10x difference.

Key open questions for investigation

1. What is the actual per-image build time on a cold-cache run? Is it the docker buildx build step that's slow, or the pruning between batches? The logs from a slow run should show this.

2. Would reverting ONLY the SDK submodule (keeping 2bfcc6c's code changes + making the get_logger import lazy) fix the builds? This would test if the SDK Dockerfile changes are truly the primary cause.

3. What is the actual sdist tarball size difference between old and new SDK? And does --extra boto3 significantly increase the Docker build context?

4. Can the BuildKit prune strategy be tuned to handle the larger per-image footprint? (e.g., higher BUILDKIT_PRUNE_KEEP_GB, or pruning less aggressively)

Suggested experiments

1. Isolate the SDK variable: Dispatch a 500-image build on current main (with 2bfcc6c) but override sdk-commit to the OLD SDK (b498a699). If this completes in ~1 hour, it confirms the SDK Dockerfile changes are the root cause.

2. Isolate the get_logger variable: On a test branch, remove only the from openhands.sdk import get_logger line from image_utils.py (replace with stdlib logging). Dispatch a 500-image build with the NEW SDK. If it's still slow, fork+threading is not the issue.

3. Isolate the Dockerfile variable: On the SDK side, test a build where only the --extra boto3 is removed (keeping Python 3.12). This would tell us whether boto3 dependencies are the disk footprint driver.

4. Check prune timing: In the logs of a slow run, measure time spent in docker buildx prune vs actual docker buildx build. If prune dominates, the fix is in the prune strategy, not the build code.

[simonrosenberg]

After digging through the issue thread, PRs #456 / #486 / #487 / #499 / #503, the benchmarks and software-agent-sdk code, and representative fast/slow workflow logs, here is the plan I would use to get to a real fix instead of another partial mitigation.

Current read of the problem

2bfcc6c bundled two unrelated changes:

1. a useful benchmarks refactor (create_docker_workspace(), ensure_local_image(), local_image_exists())
2. an SDK submodule bump from b498a699 to bde715c1

For the 500-image build path, the dominant hot path is:

• benchmarks/utils/build_utils.py::build_all_images()
• benchmarks/utils/build_utils.py::build_image()
• vendor/software-agent-sdk/.../openhands/agent_server/docker/build.py::build()
• vendor/software-agent-sdk/.../openhands/agent_server/docker/build.py::_make_build_context()

My current conclusion is:

• Primary issue: the SDK bump invalidated / amplified BuildKit layer caching enough that cold-cache runs balloon to ~600 GiB and then spend huge time in prune/rebuild cycles.
• Secondary issue: per-image uv build --sdist is real overhead, but not the main 5-hour regression.
• Secondary correctness bug: importing the SDK logger before ProcessPoolExecutor forks is risky and should be fixed, but it does not match the observed “slow but progressing” behavior.

Why I think this

The strongest evidence is from the runs themselves:

• Fast historical full build: run 20806796288 on January 8, 2026

  • 500 builds / 0 skips
  • batch 19 BuildKit usage: 55.41 GiB / 655.46 GiB
  • total time: 52m

• Slow run on 2bfcc6c: run 22694639473 on March 4, 2026

  • 285 builds before cancel
  • batch 19 BuildKit usage: 598.93 GiB / 930.08 GiB
  • at batch 19 it finally crosses threshold and starts prune
  • total elapsed before cancel: ~5h49m

• Slow run with #499 sdist reuse already active: run 22929182421 on March 10-11, 2026

  • logs show Using pre-built sdist ...
  • still reaches 551.25 GiB / 911.38 GiB by batch 21
  • still takes 4h+ to get there

That last point is especially important: it means #499 was helping the sdist problem, but the dominant slowdown remained.

Likely regression inside the SDK bump

The SDK bump range includes the changes most likely to explain the cache-footprint jump:

• 97731fe5: revert agent-server Docker image to Python 3.12
• 5d65f389: include boto3 extra in agent-server Docker image
• plus the already-existing per-image clean build-context generation in _make_build_context()

Those changes affect the Dockerfile dependency graph and builder layers, which is exactly where the huge warm-cache vs cold-cache split shows up.

What I do not think is the main cause

• PR Fix fork+threading deadlock in SWE-Bench image builds #486 (spawn): addresses fork safety, not the 10x disk growth.
• PR Disable rich logging by default to fix multiprocessing deadlock #487 (LOG_JSON=1): addresses logging / RichHandler behavior, not the 10x disk growth.
• PR fix: Reuse pre-built SDK sdist to speed up image builds #499 (sdist reuse): reduces packaging overhead, but the slow test run shows it does not fix the main regression.
• Final image tag semantics are probably not the core issue, because we have both fast and slow true 500-build / 0-skip runs. The important cache is the registry BuildKit cache, not only final image skips.

Proposed plan

Phase 0: unblock production use

1. Merge PR Revert "feat: auto-detect pre-built Docker images across all benchmarks (#456)" #503 if the revert testing confirms it restores healthy build times.
   • This is the right short-term move.
   • The refactor can be reintroduced later without the SDK bump.

Phase 1: instrument before guessing more

2. Add temporary timing instrumentation behind an env flag so we can quantify where time goes.

   • Around _make_build_context() in SDK build.py
   • Around the docker buildx build call in SDK build.py
   • Around maybe_prune_buildkit_cache() in benchmarks/utils/build_utils.py
   • Optionally record docker buildx du --verbose after each batch

   We need actual timing split between:

   • clean build context creation
   • docker buildx build
   • docker buildx prune

   Right now most conclusions are still inferred from coarse logs.

Phase 2: run a clean isolation matrix

3. Run a 2x2 experiment matrix with isolated cache/image namespaces.

   Previous runs are confounded by shared GHCR cache state. For each experiment, use a unique --image repo/name so cache-from / cache-to do not contaminate one another.

   Matrix:

   • A: pre-2bfcc6c benchmarks + old SDK b498a699
   • B: 2bfcc6c benchmarks code + old SDK b498a699
   • C: pre-2bfcc6c benchmarks + new SDK bde715c1
   • D: full 2bfcc6c + new SDK bde715c1

   For each cell, run:

   • first run: cold cache
   • second run: warm cache on same image namespace

   This answers three questions cleanly:

   • Is the slowdown caused by the benchmarks refactor?
   • Is it caused by the SDK bump?
   • Does the variant warm up successfully, or does prune keep preventing reuse?

Phase 3: bisect the SDK hot path if SDK is confirmed

4. If the matrix shows the SDK is the real culprit, test these exact historical SDK checkpoints, not ad hoc local edits:

   • b498a699 baseline
   • 97731fe5 Python 3.12 Dockerfile change
   • 5d65f389 add --extra boto3
   • bde715c1 final SDK ref bundled in 2bfcc6c

   Same benchmark commit, same workflow inputs, isolated image namespace.

   This should tell us whether the main jump comes from:

   • Python 3.12 change
   • --extra boto3
   • or the combination

Phase 4: fix the real regression, not just the symptoms

5. Decouple the refactor from the SDK bump.

   • Reintroduce the benchmarks helper refactor in a PR without bumping the SDK submodule.
   • Keep that change small and easy to verify.

6. Fix fork safety in benchmarks utilities independently.

   • Replace from openhands.sdk import get_logger in utility modules with stdlib logging in:
     • benchmarks/utils/image_utils.py
     • benchmarks/utils/build_utils.py
     • any similar utility modules used before ProcessPoolExecutor forks
   • This removes a real correctness hazard even if it is not the main slowdown.

7. If the SDK Dockerfile is the culprit, fix it in the SDK build profile.
   This is where I think the real fix probably belongs.

   Candidate directions:

   • Make benchmark-oriented source-minimal builds use a lighter dependency profile.
   • Avoid installing extras such as boto3 unless they are actually needed for that target.
   • Scope the Python-version workaround only to the targets/modes that require it.

   In other words: the fix should likely reduce the layer footprint of the SDK’s benchmark build path, not just tune pruning harder.

8. Only after the main regression is fixed, keep the useful optimization from #499.

   • Reusing a prebuilt sdist is still a good idea.
   • But it should be treated as a follow-up optimization, not the root-cause fix.
   • Longer term, it would be better as an SDK-supported option than a benchmarks-side monkeypatch.

Phase 5: operational mitigations

9. Add workflow protections so bad runs fail faster and are easier to reason about:
   • timeout-minutes on the build job / step
   • explicit logging of prune duration
   • for intentionally cold-cache SDK bumps, use a larger BuildKit budget or larger-disk runner for the first seed run
   • optionally run a small warm-up build before the full 500-image build when changing the SDK Dockerfile significantly

What I would run next, in order

If we want the fastest path to certainty, I would do these runs in this order:

1. PR Revert "feat: auto-detect pre-built Docker images across all benchmarks (#456)" #503 revert branch

   • confirm that build performance is restored

2. 2bfcc6c with sdk-commit=b498a699

   • if this is fast, the benchmarks refactor is exonerated

3. revert branch / pre-2bfcc6c benchmarks with sdk-commit=bde715c1

   • if this is slow, the SDK bump is confirmed as the main regression

4. SDK historical checkpoints

   • 97731fe5
   • 5d65f389
   • compare to b498a699 and bde715c1

That sequence should tell us quickly whether the true fix belongs in:

• benchmarks refactor
• SDK Dockerfile / dependency profile
• or only the surrounding prune/cache policy

If helpful, I can turn this into a concrete dispatch matrix with exact workflow inputs and expected interpretations for each run.

[simonrosenberg]

Refinements to the plan above

The plan is solid. Here are four refinements I'd suggest after digging through the SDK code and testing cache tag behavior:

1. Add 35d75e3a to the Phase 3 SDK bisection

I traced through the _base_slug() truncation logic changed in SDK commit 35d75e3a and found it changes the actual buildcache-* tag values for a subset of SWE-bench images — specifically those with long slugified names (matplotlib, scikit-learn, sphinx-doc, etc.).

For example, docker.io/swebench/sweb.eval.x86_64.matplotlib__matplotlib-22711:latest:

• Old SDK slug: sweb.eval.x86_64.matplotlib__matplotlib-22711_tag_l-9cf33e44b735
• New SDK slug: sweb.eval.x86_64.matplotlib__matplotlib-_tag_latest-9cf33e44b735

The old code splits on the first _tag_ and truncates the tag. The new code (rsplit) splits on the last _tag_ and truncates the repo instead. This means --cache-from type=registry lookups fail for these images even if the cached layers were otherwise compatible.

The Phase 3 bisection should test 5 checkpoints:

1. b498a699 (baseline)
2. 35d75e3a (cache tag truncation)
3. 97731fe5 (Python 3.12)
4. 5d65f389 (--extra boto3)
5. bde715c1 (final)

This is cheap to test and would reveal whether 35d75e3a alone causes partial cache misses — a bug that could be fixed independently in the SDK.

2. Trim the Phase 2 matrix to save runner cost

Running 8 full 500-image builds (4 cells × cold + warm) is expensive. We already know from historical data that old SDK warms up fine (run 20806796288: 500 builds, 55 GiB, 52 min). I'd skip warm-cache runs for cells A and B (old SDK) and only run warm-cache for cells C and D (new SDK). The question for C/D is: can the new SDK ever warm up successfully, or does the prune cycle prevent it?

3. Elevate "make cold-cache survivable" to equal status with "reduce SDK layer footprint"

Both --extra boto3 and Python 3.12 were added for real reasons (functionality and PyInstaller+libtmux compat per issue #1886). If the bisection confirms both are necessary, "reduce footprint" isn't viable.

Any significant SDK Dockerfile change will produce a cold-cache event. The infrastructure should handle it regardless:

• Larger-disk runner or higher BUILDKIT_PRUNE_KEEP_GB (150-200) for the first "seed" run after an SDK Dockerfile change
• Reduced max-workers for cold-cache runs to limit concurrent disk pressure
• After the seed run writes warm cache to the registry, subsequent runs use normal settings

I'd treat this as co-equal with the SDK Dockerfile fix (Phase 4), not just an operational mitigation (Phase 5), because cold-cache runs will recur with future SDK changes.

4. Parallelize the first experiments

Runs 1–3 from the execution order are independent. With isolated --image namespaces they can be dispatched concurrently while SDK bisection branches are being prepared.