Skip to content

execStream can end with a fabricated 'complete' (exitCode 0) while the process is still running (0.10.3) #873

Description

@AbdallahAbuHashem

Summary

On @cloudflare/sandbox 0.10.3 (container image cloudflare/sandbox:0.10.3, standard-4 instances), long-running exec-streamed commands intermittently have their SSE stream end with a complete event carrying exitCode: 0while the underlying process is still alive and running inside the container.

The caller sees a clean, successful-looking stream end mid-command, loses all subsequent output, and has no signal that anything went wrong.

Evidence

We run an agent CLI (long-lived, minutes-scale, emits JSONL to stdout) via a Worker route that returns sandbox.execStream(cmd, { cwd, env, timeout }) to the client. During a burst of these events we probed the affected sandboxes immediately after the "complete" was received:

  • pgrep in the same sandbox showed the process still running (in 2 of 3 probed cases), minutes after its stream had "completed" with exit 0.
  • The process's output continued to grow (we redirect a transcript to disk) — the work eventually finished, but the result had streamed into a void.
  • The final SSE frames before the fabricated complete are ordinary stdout mid-flow (often mid-line of tool output), then {"type":"complete","exitCode":0} with no error event and nothing on stderr.

Frequency for us: dozens/day across sessions, and it correlates with heavier turns (long duration, high stdout volume, subprocess trees) — short commands almost never hit it.

What we could NOT reproduce it with (all pass cleanly through the same route)

  • 3.6 MB of fast line-oriented output through exec/stream
  • a single 3 MB line (no newline until the end)
  • concurrent exec calls against the same sandbox while a stream is open
  • createBackup running concurrently with an open stream
  • long --resume invocations of the same CLI via non-streaming exec

So plain volume / concurrency / big-line cases are fine; the trigger appears to be something about real long-lived processes (possibly DO lifecycle events — replacement/restart — during the stream: severed streams with no complete event also occur in the same windows, and those cluster across many sandboxes simultaneously).

Expected

If the runtime loses track of the process or the stream is interrupted (DO restart, container replacement, transport loss), the stream should end with an error / structured interruption, never a fabricated complete with exitCode 0. (The 0.12.3 changelog — "structured errors for sandbox runtime and platform interruptions" — sounds related; can you confirm whether this failure mode is fixed in 0.12.x?)

Environment

  • @cloudflare/sandbox 0.10.3, HTTP transport, Workers route returning the raw SSE
  • Container image based on cloudflare/sandbox:0.10.3, standard-4, sleepAfter: 30m, keepAlive: true
  • Observed 2026-08-16 → 2026-08-17, multiple sandboxes/DOs

Happy to provide timestamps/ray IDs privately if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions