`inflightRequests` leaks on client-aborted proxied requests — container never stops and bills indefinitely

[noorderling-nl]

Summary

When a client aborts a request while it is being proxied to the container, the inflightRequests counter leaks (it is incremented but never decremented). From that moment on, isActivityExpired() always returns false, so onActivityExpired() is never called, the container never receives SIGTERM, and the instance keeps running and billing indefinitely. The leak is self-sustaining: the recurring alarm keeps the Durable Object pinned in memory, so the (in-memory) counter is never reset by DO eviction either.

We have hit this twice in production: one instance ran for 17.5 hours (2026-08-09) and one for 79 minutes (2026-08-14, caught early by a billing watchdog) with zero traffic after the initial burst, on sleepAfter = '2m'.

Affected versions

Verified in the published source of both 0.2.4 and 0.3.7 (latest at the time of writing). The relevant code is identical in both.

Mechanism

In containerFetch (dist/lib/container.js, 0.3.7 line numbers):

this.inflightRequests++;                                  // L887
try {
    this.renewActivityTimeout();
    const res = await tcpPort.fetch(containerUrl, request); // L891
    ...
} catch (e) {
    this.decrementInflight();                             // L962
    ...

If the incoming request is cancelled while await tcpPort.fetch(...) is pending (client disconnect/abort — in observability these invocations show outcome: "canceled"), the invocation's continuations never run: neither the success path nor the catch executes, so decrementInflight() is never called. The increment on L887 already happened synchronously. Net effect: inflightRequests is permanently ≥ 1.

Then in the alarm path:

isActivityExpired() {                                     // L1687
    if (this.inflightRequests > 0) {
        this.renewActivityTimeout();
        return false;
    }
    return this.sleepAfterMs <= Date.now();
}

Every alarm sees a non-zero counter, renews the deadline by sleepAfter, and reschedules. The container is never told to stop, and the alarm cadence itself keeps the DO alive in memory, so the leaked counter never resets. The only way out is external (we used a same-image rolling rollout via the API to force the instance to exit).

Observed behaviour (production, 2026-08-14)

• 12:20:19Z: burst of ~30 concurrent requests proxied to one container DO during cold start; several show outcome: "canceled" in Workers observability.
• 12:20:20Z: last request completes. No traffic to this DO after this point.
• From 12:22 onward: one alarm invocation every 120.06s (= sleepAfter 2m), outcome: "ok", forever. The "Activity expired, signalling container to stop" log line never appears for this DO (sibling DOs from the same deployment that had no cancelled requests logged it and stopped normally).
• Instance state stayed running (confirmed via GET /accounts/{acc}/containers/applications/{app}/instances) until we forced a rollout at 13:38Z.

Repro sketch

1. class MyContainer extends Container { sleepAfter = '2m'; }, proxy requests via getContainer(...).fetch(...).
2. Fire a request at a cold DO and abort it client-side while the container is still starting (before the proxied fetch resolves).
3. Observe: the DO alarm fires every sleepAfter interval indefinitely, onActivityExpired is never invoked, and the container instance keeps running (and billing) with zero traffic.

Suggested direction

inflightRequests needs to be robust against invocation cancellation, e.g.:

• decrement from a request.signal abort listener (if the runtime delivers it in this situation), and/or
• treat the counter as advisory in isActivityExpired(): cap how long a non-zero counter may defer expiry without any new renewActivityTimeout() from real traffic (a leaked counter currently defers it forever), and reconcile/zero it at that bound.

Related but distinct: #162 (alarm firing during a long containerFetch) — this issue is the mirror image: the alarm never firing the expiry because a cancelled fetch left the counter stuck.

Workaround

We now run an idle self-exit watchdog inside the container image (exit 0 after N seconds with no requests and no queued work), which bounds the damage regardless of DO-side accounting.

Happy to share account/instance IDs privately for billing-side verification.