We hit six bugs in the local-dev bucket mount path (mountBucket(..., { localBucket: true })) while running an agent workload against wrangler dev with local containers, where /dev/fuse is unavailable and local sync is the only way to get an archive in or out of a sandbox. Together they make local mounts unusable for anything larger than a few MB, and permanently broken after a Durable Object restart.
A complete fix with tests exists on a public branch, one commit on top of main (32e49a2):
This repository limits opening pull requests to collaborators, so I am filing the analysis here instead. A maintainer is welcome to pull the branch directly, or invite the PR and I will open it immediately.
The same six fixes, applied as a patch over the published 0.12.4 bundle, are what we run locally today: a full end-to-end (baseline prep → task creation → agent turn → 13.8 MB checkpoint restore through the mount → second turn) passes with them and fails without.
Environment: @cloudflare/sandbox 0.12.4 (analysis and branch verified against main, 0.12.5), SANDBOX_TRANSPORT=rpc, wrangler dev local containers, localBucket: true mounts.
Each section below pairs one bug with what the branch does about it.
Repro
An R2 bucket holding a 13.8 MB checkpoint archive, mounted for local dev:
await sandbox.mountBucket('WORKSPACE', '/workspace', { localBucket: true });The initial R2 → container sync kills the RPC session:
OperationInterruptedError: Sandbox operation files.writeFile was interrupted
while the runtime connection was closing (kind: session_disposed)
caused by: RPC session was shut down by disposing the main stub
With a ~1 GB object the same write fails earlier, in the Worker:
RangeError: String is too long for a V8 string
at uint8ArrayToBase64 (packages/sandbox/src/local-mount-sync.ts)
Writing a large file into the mount fails the same way on its way back to R2. Calling mountBucket() again for the same bucket and path — which callers do because they cannot tell whether a warm sandbox already has the mount — throws InvalidMountConfigError: Mount path already in use. And after the Worker restarts (or the DO is evicted while the container idles out), every mount attempt fails with the session_disposed interruption above and never recovers.
1. A base64 write cannot exceed one control frame
transferR2ObjectToContainer() does bucket.get() → arrayBuffer() → Buffer.toString('base64') → files.writeFile(..., { encoding: 'base64' }).
On the rpc transport that payload rides a single WebSocket frame, and the container control server caps frames at 16 MiB (Bun's default maxPayloadLength). Base64 inflates content by 4/3, so anything over ~12 MB overruns the frame; the control socket closes, and capnweb surfaces it as RPC session was shut down by disposing the main stub, failing every in-flight and subsequent call on that session. This is the limit users hit first — long before the V8 string limit, which stops base64 working at all a few hundred MB later.
Fix: objects above 4 MiB stream straight into the container via files.writeFileStream(path, obj.body, sessionId) on the rpc transport — the same approach doRestoreBackupLocal() already uses for backup archives. 4 MiB keeps the base64 path well inside the frame budget. Below the threshold, and on the http and websocket transports (which have no stream write), the base64 path is unchanged.
2. Container → R2 reads the whole file as one base64 string
uploadFileToR2() reads the file with readFile(..., { encoding: 'base64' }) and put()s the decoded bytes — the whole file in Worker memory, and past the V8 string limit no upload at all.
Fix: read with readFileStream() through the existing streamFile() SSE decoder and accumulate exactly 16 MiB parts. A file larger than one part goes up as an R2 multipart upload; smaller files keep the single put(). Two details worth a look while reviewing:
- Parts are sliced to an exact size (
takePart()) rather than left chunk-aligned, because R2 rejects a multipart upload whose non-final parts differ in size.
- A failing part aborts the upload instead of leaving its uploaded parts behind in the bucket.
The post-upload head() snapshot refresh is unchanged, so echo suppression against the R2 poll loop still works.
3. Concurrent transfers share one RPC session
Both R2 → container loops (fullSyncR2ToContainer() and pollR2ForChanges()) run Promise.all batches of SYNC_CONCURRENCY transfers. A streamed transfer holds a stream export open on the shared capnweb session; when a sibling in the batch rejects, Promise.all abandons that export mid-flight, the session is torn down, and every other call on it fails — including the rest of the sync.
Fix: both loops transfer one object at a time. Error attribution also becomes unambiguous: a failure names the object it belongs to. If you would rather keep concurrency for objects below the streaming threshold, that is easy to add on top — the branch goes with sequential as the safe default.
4. Every inotify event uploads the whole file
runContainerWatchLoop() uploads on every create/modify/move_to. One large file being written into the mount (tar -x, cp) emits a burst of modify events, so that file is uploaded again and again, each time in a partial state.
Fix: a trailing 1.5 s per-path debounce, so the last event of a burst wins and each settled file is uploaded once. delete/move_from cancels a queued upload before deleting the object, so a queued upload cannot resurrect a deleted file. stop() clears pending timers and reports how many it dropped (droppedUploads in the stop log): once the watch stream is gone, nothing can tell whether those files finished being written. Uploads had no completion guarantee across stop() before this change either — the watch loop was abandoned mid-await.
5. The default session outlives the container that created it
ensureDefaultSession() returns the cached defaultSession on a bare id match. But defaultSession is restored from durable storage on cold start, while containerGeneration is memory-only and restarts at 0 — so a Durable Object evicted while its container was replaced comes back holding a session id the new container runtime never created. Every session-scoped call then fails with OperationInterruptedError (session_disposed), and because nothing invalidates the cached id, the sandbox never recovers. Mounts are simply where this shows up first: LocalMountSyncManager does all of its file I/O on the default session.
Fix: the cache carries the generation it was created against (defaultSessionGeneration, initially -1), and both the fast path and the invalidated-init retry require it to match containerGeneration. A session id restored from storage therefore costs exactly one re-initialization per DO instance, and against a still-live container that re-init is a no-op through the existing SessionAlreadyExistsError path.
6. Local mounts cannot be re-mounted
mountBucketLocal() throws Mount path already in use whenever activeMounts holds the path. But a local-sync mount is a plain directory, so mountpoint -q inside the container never reports it as mounted; the probe-then-mount pattern that works for FUSE mounts therefore re-mounts on every call and fails on the second one against a warm sandbox.
Fix: a repeat mount of the same bucket, prefix and readOnly returns successfully without starting a second sync loop — provided the existing mount belongs to the current container generation. If it belongs to an earlier one its sync manager holds a dead session, so it is stopped, dropped, and replaced with a fresh mount against the live container. Anything else — a different bucket, prefix or readOnly, or a mount of another type — still throws. LocalSyncMountInfo carries prefix, readOnly and containerGeneration for those comparisons, mirroring R2BindingMountInfo.
What the branch contains
No public API change. 18 new unit tests (13 added to tests/local-mount-sync.test.ts, 4 in a new tests/local-mount.test.ts, 1 in tests/sandbox.test.ts) and a patch changeset. Each new test was checked to fail without its corresponding source change. npm run check passes, and npm test -w @cloudflare/sandbox is green at 52 files / 1097 tests. E2E was not run there (it needs the container image built), though the equivalent patch is exercised end-to-end in our own stack as described above.
Behaviour on the http and websocket transports is unchanged: writeFileStream is rpc-only, so those transports keep the base64 write and its size ceiling. Small files keep the single put(), conflicting mounts still throw, and the session change costs one idempotent createSession per DO instance when a session id was restored from storage.
Happy to split this into separate PRs (bug 5 is not mount-specific — it affects every consumer of the default session), rework any of it to your taste, or just leave the branch here for someone to cherry-pick.
We hit six bugs in the local-dev bucket mount path (
mountBucket(..., { localBucket: true })) while running an agent workload againstwrangler devwith local containers, where/dev/fuseis unavailable and local sync is the only way to get an archive in or out of a sandbox. Together they make local mounts unusable for anything larger than a few MB, and permanently broken after a Durable Object restart.A complete fix with tests exists on a public branch, one commit on top of
main(32e49a2):This repository limits opening pull requests to collaborators, so I am filing the analysis here instead. A maintainer is welcome to pull the branch directly, or invite the PR and I will open it immediately.
The same six fixes, applied as a patch over the published 0.12.4 bundle, are what we run locally today: a full end-to-end (baseline prep → task creation → agent turn → 13.8 MB checkpoint restore through the mount → second turn) passes with them and fails without.
Environment:
@cloudflare/sandbox0.12.4 (analysis and branch verified againstmain, 0.12.5),SANDBOX_TRANSPORT=rpc,wrangler devlocal containers,localBucket: truemounts.Each section below pairs one bug with what the branch does about it.
Repro
An R2 bucket holding a 13.8 MB checkpoint archive, mounted for local dev:
The initial R2 → container sync kills the RPC session:
With a ~1 GB object the same write fails earlier, in the Worker:
Writing a large file into the mount fails the same way on its way back to R2. Calling
mountBucket()again for the same bucket and path — which callers do because they cannot tell whether a warm sandbox already has the mount — throwsInvalidMountConfigError: Mount path already in use. And after the Worker restarts (or the DO is evicted while the container idles out), every mount attempt fails with thesession_disposedinterruption above and never recovers.1. A base64 write cannot exceed one control frame
transferR2ObjectToContainer()doesbucket.get()→arrayBuffer()→Buffer.toString('base64')→files.writeFile(..., { encoding: 'base64' }).On the rpc transport that payload rides a single WebSocket frame, and the container control server caps frames at 16 MiB (Bun's default
maxPayloadLength). Base64 inflates content by 4/3, so anything over ~12 MB overruns the frame; the control socket closes, and capnweb surfaces it asRPC session was shut down by disposing the main stub, failing every in-flight and subsequent call on that session. This is the limit users hit first — long before the V8 string limit, which stops base64 working at all a few hundred MB later.Fix: objects above 4 MiB stream straight into the container via
files.writeFileStream(path, obj.body, sessionId)on the rpc transport — the same approachdoRestoreBackupLocal()already uses for backup archives. 4 MiB keeps the base64 path well inside the frame budget. Below the threshold, and on the http and websocket transports (which have no stream write), the base64 path is unchanged.2. Container → R2 reads the whole file as one base64 string
uploadFileToR2()reads the file withreadFile(..., { encoding: 'base64' })andput()s the decoded bytes — the whole file in Worker memory, and past the V8 string limit no upload at all.Fix: read with
readFileStream()through the existingstreamFile()SSE decoder and accumulate exactly 16 MiB parts. A file larger than one part goes up as an R2 multipart upload; smaller files keep the singleput(). Two details worth a look while reviewing:takePart()) rather than left chunk-aligned, because R2 rejects a multipart upload whose non-final parts differ in size.The post-upload
head()snapshot refresh is unchanged, so echo suppression against the R2 poll loop still works.3. Concurrent transfers share one RPC session
Both R2 → container loops (
fullSyncR2ToContainer()andpollR2ForChanges()) runPromise.allbatches ofSYNC_CONCURRENCYtransfers. A streamed transfer holds a stream export open on the shared capnweb session; when a sibling in the batch rejects,Promise.allabandons that export mid-flight, the session is torn down, and every other call on it fails — including the rest of the sync.Fix: both loops transfer one object at a time. Error attribution also becomes unambiguous: a failure names the object it belongs to. If you would rather keep concurrency for objects below the streaming threshold, that is easy to add on top — the branch goes with sequential as the safe default.
4. Every inotify event uploads the whole file
runContainerWatchLoop()uploads on everycreate/modify/move_to. One large file being written into the mount (tar -x,cp) emits a burst ofmodifyevents, so that file is uploaded again and again, each time in a partial state.Fix: a trailing 1.5 s per-path debounce, so the last event of a burst wins and each settled file is uploaded once.
delete/move_fromcancels a queued upload before deleting the object, so a queued upload cannot resurrect a deleted file.stop()clears pending timers and reports how many it dropped (droppedUploadsin the stop log): once the watch stream is gone, nothing can tell whether those files finished being written. Uploads had no completion guarantee acrossstop()before this change either — the watch loop was abandoned mid-await.5. The default session outlives the container that created it
ensureDefaultSession()returns the cacheddefaultSessionon a bare id match. ButdefaultSessionis restored from durable storage on cold start, whilecontainerGenerationis memory-only and restarts at 0 — so a Durable Object evicted while its container was replaced comes back holding a session id the new container runtime never created. Every session-scoped call then fails withOperationInterruptedError(session_disposed), and because nothing invalidates the cached id, the sandbox never recovers. Mounts are simply where this shows up first:LocalMountSyncManagerdoes all of its file I/O on the default session.Fix: the cache carries the generation it was created against (
defaultSessionGeneration, initially-1), and both the fast path and the invalidated-init retry require it to matchcontainerGeneration. A session id restored from storage therefore costs exactly one re-initialization per DO instance, and against a still-live container that re-init is a no-op through the existingSessionAlreadyExistsErrorpath.6. Local mounts cannot be re-mounted
mountBucketLocal()throwsMount path already in usewheneveractiveMountsholds the path. But a local-sync mount is a plain directory, somountpoint -qinside the container never reports it as mounted; the probe-then-mount pattern that works for FUSE mounts therefore re-mounts on every call and fails on the second one against a warm sandbox.Fix: a repeat mount of the same bucket, prefix and
readOnlyreturns successfully without starting a second sync loop — provided the existing mount belongs to the current container generation. If it belongs to an earlier one its sync manager holds a dead session, so it is stopped, dropped, and replaced with a fresh mount against the live container. Anything else — a different bucket, prefix orreadOnly, or a mount of another type — still throws.LocalSyncMountInfocarriesprefix,readOnlyandcontainerGenerationfor those comparisons, mirroringR2BindingMountInfo.What the branch contains
No public API change. 18 new unit tests (13 added to
tests/local-mount-sync.test.ts, 4 in a newtests/local-mount.test.ts, 1 intests/sandbox.test.ts) and a patch changeset. Each new test was checked to fail without its corresponding source change.npm run checkpasses, andnpm test -w @cloudflare/sandboxis green at 52 files / 1097 tests. E2E was not run there (it needs the container image built), though the equivalent patch is exercised end-to-end in our own stack as described above.Behaviour on the http and websocket transports is unchanged:
writeFileStreamis rpc-only, so those transports keep the base64 write and its size ceiling. Small files keep the singleput(), conflicting mounts still throw, and the session change costs one idempotentcreateSessionper DO instance when a session id was restored from storage.Happy to split this into separate PRs (bug 5 is not mount-specific — it affects every consumer of the default session), rework any of it to your taste, or just leave the branch here for someone to cherry-pick.