Summary
Repeated one-shot Python executions substantially raise the shared native sidecar's proportional set size (PSS). Waiting for each execution, disposing its VM, deleting its persisted workspace, and waiting five seconds does not return the sidecar close to its cold memory footprint. Exiting the sidecar process does.
A fresh-process staged probe suggests a retained/high-watermark behavior rather than enough evidence for an unbounded linear leak:
- cold sidecar: ~18 MiB PSS
- standard-library Python, after 5 isolated executions: ~270 MiB
- NumPy, after 8 additional isolated executions: ~422 MiB
- pandas, across 8 additional isolated executions: ~445–558 MiB, non-monotonic
The pandas stage sometimes drops by ~80 MiB, so this issue intentionally does not claim that every execution permanently leaks a fixed amount. The operational problem is that VM disposal is not currently a reliable physical-memory reclamation boundary for one-shot Pyodide workloads in a long-lived shared sidecar.
Environment
@rivet-dev/agentos: 0.2.16-rc.1- Node.js:
v22.22.2
- OS: Linux x86_64, kernel
6.6.98
- glibc:
2.38
- sidecar placement: shared process
- Python mode: one-shot
vm.python.spawn, not retained contexts
- packages: Pyodide-provided NumPy and pandas assets
Probe procedure
The probe starts a fresh Runtime/native sidecar and records Linux PSS from /proc/<sidecar-pid>/smaps_rollup.
For every iteration it:
- creates a new VM/workspace;
- calls
vm.python.spawn(...);
- waits for the execution to finish;
- disposes the VM;
- deletes the workspace SQLite files;
- waits 5 seconds;
- records sidecar PSS.
Workloads are executed sequentially in this order:
cold baseline
standard library x5
NumPy x8
pandas x8
Each execution uses a distinct VM. There are no retained Python contexts.
Results
PSS values are KiB:
baseline 0 18,374
stdlib 1 241,806
stdlib 2 263,672
stdlib 3 269,970
stdlib 4 270,846
stdlib 5 276,930
numpy 1 356,644
numpy 2 372,698
numpy 3 397,170
numpy 4 412,674
numpy 5 413,298
numpy 6 416,974
numpy 7 420,218
numpy 8 421,774
pandas 1 558,232
pandas 2 527,151
pandas 3 444,790
pandas 4 461,963
pandas 5 476,763
pandas 6 486,619
pandas 7 539,143
pandas 8 461,629
Execution times were approximately 19–23 seconds for standard-library/NumPy runs and 26–30 seconds for pandas runs.
A separate already-warm probe started around 971 MiB PSS and reached approximately 1,162 MiB after three NumPy+pandas executions, despite deleting each workspace and waiting after each run. Restarting the Runtime/sidecar returned it to approximately 31 MiB.
File descriptors returned to the baseline range, and deleting SQLite/workspace files did not lower PSS. The memory belongs to the native sidecar process rather than the Node host process.
Expected behavior / requested contract
Returning exactly to cold PSS after every execution may be unrealistic for V8, Pyodide, WebAssembly memories, and the native allocator. What is needed for a long-lived shared sidecar is one of:
- repeated identical one-shot workloads reach a documented, bounded plateau after VM/context disposal; or
- AgentOS exposes a supported reclamation/lifecycle boundary that releases Python runtime memory without terminating unrelated VMs in the shared sidecar.
At minimum, it would be useful to document the expected memory lifecycle and recommended production strategy.
Suspected boundary
Python runs inside the shared sidecar approximately as:
native sidecar
-> V8 session
-> Pyodide
-> WebAssembly linear memory / ArrayBuffer backing stores
-> NumPy / pandas
AgentOS has context/VM disposal paths and removes logical tracking state, but logical disposal does not demonstrate that V8 sessions, WASM backing stores, or allocator arenas are released or returned to Linux.
Possible causes include:
- disposed V8/Pyodide objects still being referenced;
- delayed collection of WASM/ArrayBuffer backing stores;
- native allocator arenas retaining released pages;
- a combination of these.
The non-monotonic pandas measurements make allocator/runtime high-watermark retention the leading explanation, but a longer 30–100 iteration low-point slope is needed to rule out an unbounded component.
Potentially relevant source areas:
crates/execution/src/python.rscrates/execution/src/javascript.rscrates/execution/assets/runners/python-runner.mjscrates/native-sidecar/tests/python.rs
The Python integration tests also note shared-process teardown/init instability for V8/Pyodide-backed cases and process-global Pyodide module state, which may be relevant to defining the supported lifecycle.
Suggested regression / diagnostics
A useful upstream regression would run in a dedicated sidecar OS process:
cold start
standard library x10
NumPy x30
pandas x30
For every iteration:
- wait for one-shot execution completion;
- dispose context and VM;
- wait a fixed settle period;
- record PSS/RSS/private dirty, active VM/process/context counts, and V8/WASM memory metrics.
After discarding package materialization/warm-up iterations, evaluate the slope of local PSS minima rather than requiring memory to return to cold baseline. Heap/allocation profiling could then distinguish reachable V8 sessions, retained WASM backing stores, Rust/JS tracking objects, and allocator-only retention.
Why this matters
A shared sidecar can serve many independent VMs. If the reliable physical reclamation boundary is the sidecar process rather than the VM, operators need to drain and restart the entire sidecar based on memory thresholds, affecting unrelated sessions. A bounded per-VM or Python-runtime reclamation boundary would avoid that operational coupling.
Summary
Repeated one-shot Python executions substantially raise the shared native sidecar's proportional set size (PSS). Waiting for each execution, disposing its VM, deleting its persisted workspace, and waiting five seconds does not return the sidecar close to its cold memory footprint. Exiting the sidecar process does.
A fresh-process staged probe suggests a retained/high-watermark behavior rather than enough evidence for an unbounded linear leak:
The pandas stage sometimes drops by ~80 MiB, so this issue intentionally does not claim that every execution permanently leaks a fixed amount. The operational problem is that VM disposal is not currently a reliable physical-memory reclamation boundary for one-shot Pyodide workloads in a long-lived shared sidecar.
Environment
@rivet-dev/agentos:0.2.16-rc.1v22.22.26.6.982.38vm.python.spawn, not retained contextsProbe procedure
The probe starts a fresh Runtime/native sidecar and records Linux PSS from
/proc/<sidecar-pid>/smaps_rollup.For every iteration it:
vm.python.spawn(...);Workloads are executed sequentially in this order:
Each execution uses a distinct VM. There are no retained Python contexts.
Results
PSS values are KiB:
Execution times were approximately 19–23 seconds for standard-library/NumPy runs and 26–30 seconds for pandas runs.
A separate already-warm probe started around 971 MiB PSS and reached approximately 1,162 MiB after three NumPy+pandas executions, despite deleting each workspace and waiting after each run. Restarting the Runtime/sidecar returned it to approximately 31 MiB.
File descriptors returned to the baseline range, and deleting SQLite/workspace files did not lower PSS. The memory belongs to the native sidecar process rather than the Node host process.
Expected behavior / requested contract
Returning exactly to cold PSS after every execution may be unrealistic for V8, Pyodide, WebAssembly memories, and the native allocator. What is needed for a long-lived shared sidecar is one of:
At minimum, it would be useful to document the expected memory lifecycle and recommended production strategy.
Suspected boundary
Python runs inside the shared sidecar approximately as:
AgentOS has context/VM disposal paths and removes logical tracking state, but logical disposal does not demonstrate that V8 sessions, WASM backing stores, or allocator arenas are released or returned to Linux.
Possible causes include:
The non-monotonic pandas measurements make allocator/runtime high-watermark retention the leading explanation, but a longer 30–100 iteration low-point slope is needed to rule out an unbounded component.
Potentially relevant source areas:
crates/execution/src/python.rscrates/execution/src/javascript.rscrates/execution/assets/runners/python-runner.mjscrates/native-sidecar/tests/python.rsThe Python integration tests also note shared-process teardown/init instability for V8/Pyodide-backed cases and process-global Pyodide module state, which may be relevant to defining the supported lifecycle.
Suggested regression / diagnostics
A useful upstream regression would run in a dedicated sidecar OS process:
For every iteration:
After discarding package materialization/warm-up iterations, evaluate the slope of local PSS minima rather than requiring memory to return to cold baseline. Heap/allocation profiling could then distinguish reachable V8 sessions, retained WASM backing stores, Rust/JS tracking objects, and allocator-only retention.
Why this matters
A shared sidecar can serve many independent VMs. If the reliable physical reclamation boundary is the sidecar process rather than the VM, operators need to drain and restart the entire sidecar based on memory thresholds, affecting unrelated sessions. A bounded per-VM or Python-runtime reclamation boundary would avoid that operational coupling.