Pi workers should not disable auto-compaction by mutating shared user settings

[tmustier]

Summary

Symphony's Pi runtime currently hard-disables Pi auto-compaction on every worker turn via RPC, and because Pi persists that RPC toggle to settings.json, the worker can mutate the operator's interactive Pi config.

This was introduced in the initial Pi RPC spike as a conservative simplification, but it now looks stale and actively harmful.

What I found

Original rationale

This behavior was introduced in the initial Pi worker spike:

• 3c4f677 — Add Pi RPC worker spike runner
• 635e5ac — Document the Pi RPC worker contract
• 68404eb — Add Pi worker adapter modules

The contract doc explicitly says:

Disable Pi-managed compaction for the initial spike

... compaction can be reintroduced later once multi-turn Pi worker behavior is stable

Relevant files:

• docs/PI_RPC_CONTRACT.md
• tools/pi-rpc-spike/client.ts
• orchestrator/elixir/lib/symphony_elixir/pi/rpc_client.ex

Current behavior

On each worker turn, Symphony sends:

{"type":"set_auto_retry","enabled":false}
{"type":"set_auto_compaction","enabled":false}

Today, Pi implements those RPC commands by mutating and saving global settings, rather than treating them as session-local worker toggles.

So Symphony workers can end up rewriting the operator's interactive config.

Why this is a problem

1) Workers should not mutate shared interactive Pi settings

This is the immediate bug.

If the operator's ~/.pi/agent/settings.json is shared with their interactive Pi setup (or symlinked into a tracked repo, as on my machine), Symphony worker turns can silently flip:

• compaction.enabled
• retry.enabled

That leaks worker policy into the human operator environment.

2) The original "disable auto-compaction" rationale is likely stale

Symphony now runs multi-turn Pi workers (agent.max_turns > 1 in real workflows), which is exactly the kind of unattended / long-running process that benefits from auto-compaction.

I did not find evidence of a current concrete compaction bug that still justifies hard-disabling it.

3) Re-enabling auto-compaction is not just a one-line flip

There is one real lifecycle nuance to handle correctly:

Pi can do compaction after agent_end.

• threshold compaction: agent_end -> compaction -> no auto-retry
• overflow compaction: agent_end -> compaction -> auto-retry same prompt

Current Symphony WorkerRunner treats the first agent_end as final turn completion, so simply removing set_auto_compaction false may cause turn-boundary races / desync unless the worker runner learns to tolerate post-agent_end compaction/retry.

Relevant file:

• orchestrator/elixir/lib/symphony_elixir/pi/worker_runner.ex

Proposed direction

Immediate safety fix

Do not let Pi workers share the operator's normal config dir.

When launching Pi workers, set a dedicated PI_CODING_AGENT_DIR for Symphony worker processes so any worker-scoped RPC settings cannot mutate the operator's interactive setup.

That isolates:

• settings
• auth/model metadata
• package/config side effects

from the operator's main ~/.pi/agent.

Runtime fix

Stop hardcoding:

• set_auto_compaction { enabled: false }

at least as an unconditional default.

Options:

• remove it entirely once worker lifecycle is safe
• or make it an explicit Symphony config option instead of a hardcoded spike artifact

Turn lifecycle fix

Before re-enabling compaction by default, make Pi worker turn completion robust to post-agent_end compaction.

Likely options:

• after agent_end, poll get_state until isStreaming == false and isCompacting == false
• and/or explicitly handle compaction_start / compaction_end
• and treat compaction_end.willRetry == true as "turn still in progress"

Acceptance criteria

• Symphony Pi workers no longer mutate the operator's interactive ~/.pi/agent/settings.json
• Worker retry/compaction policy is isolated from the operator's normal Pi environment
• The hardcoded set_auto_compaction false spike behavior is removed or made explicit/configurable
• Worker turn completion correctly handles post-agent_end compaction / overflow-retry semantics
• We can safely enable auto-compaction for unattended multi-turn Pi workers

Notes

Pi supports PI_CODING_AGENT_DIR explicitly, so Symphony should be able to isolate worker config without upstream Pi changes.

There is also a likely related leak for set_auto_retry, since Symphony sends that the same way and Pi persists it similarly.

[tmustier]

Supplemental confirmation from local repro: this is the real root-cause chain, not just a plausible theory.

Confirmed end-to-end

1) Pi RPC persists these toggles to disk

I ran a controlled repro with a temporary PI_CODING_AGENT_DIR and a fresh settings.json.

Sequence:

• start pi --mode rpc
• send get_state
• send set_auto_compaction { enabled: false }
• inspect temp settings.json
• send set_auto_retry { enabled: false }
• inspect temp settings.json again

Observed:

• compaction.enabled changed true -> false
• retry.enabled changed true -> false

So in Pi today, these RPC calls are persistent config writes, not session-scoped worker toggles.

2) Symphony is the only local caller I found

I searched local source trees for set_auto_compaction and the only user-authored caller I found was Symphony:

• orchestrator/elixir/lib/symphony_elixir/pi/rpc_client.ex
• tools/pi-rpc-spike/client.ts

Both send:

{"type":"set_auto_compaction","enabled":false}

and similarly for set_auto_retry.

3) Symphony workers are using the shared Pi config dir

Symphony launches Pi workers with their own --session-dir, but does not set PI_CODING_AGENT_DIR, so workers inherit the normal shared config dir (~/.pi/agent).

On my machine that file is a symlink to a tracked repo file:

• ~/.pi/agent/settings.json
• -> ~/projects/pi-setup/settings.json

So a worker turn can rewrite my interactive Pi config and also dirty a git repo.

4) Live repro with the running Symphony daemon

I re-enabled compaction.enabled, then ran a read-only monitor on ~/.pi/agent/settings.json every 5 seconds while the local Symphony daemon was active.

Observed:

• 0s through 45s: enabled=True
• 50s: enabled=False

No manual write was performed during that monitoring window.

At the same time, process inspection showed live pi worker processes parented by the Symphony VM, with cwd inside Symphony workspaces, e.g.:

• /Users/thomasmustier/symphony-workspaces/quoting-demo-v2/THO-154
• /Users/thomasmustier/symphony-workspaces/quoting-demo-v2/THO-155

That is enough to treat Symphony as the actual cause:

1. Symphony sends set_auto_compaction false / set_auto_retry false
2. Pi persists those RPC commands to settings.json
3. Symphony workers share the operator's normal config dir
4. worker policy leaks into the interactive environment

Repro steps

Repro A — prove Pi-side persistence safely

Use a temp config dir:

python3 - <<'PY'
import json, os, pathlib, subprocess, tempfile, time

agent_dir = pathlib.Path(tempfile.mkdtemp(prefix='pi-agent-dir-'))
session_dir = pathlib.Path(tempfile.mkdtemp(prefix='pi-session-dir-'))
settings = agent_dir / 'settings.json'
settings.write_text(json.dumps({
    'compaction': {'enabled': True, 'reserveTokens': 16384, 'keepRecentTokens': 20000},
    'retry': {'enabled': True}
}, indent=2))

env = os.environ.copy()
env['PI_CODING_AGENT_DIR'] = str(agent_dir)

proc = subprocess.Popen(
    ['pi', '--mode', 'rpc', '--session-dir', str(session_dir), '--no-extensions', '--no-themes'],
    stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    text=True, bufsize=1, env=env,
)

def request(payload):
    proc.stdin.write(json.dumps(payload) + '\n')
    proc.stdin.flush()
    deadline = time.time() + 10
    while time.time() < deadline:
        line = proc.stdout.readline()
        if not line:
            continue
        obj = json.loads(line)
        if obj.get('type') == 'response' and obj.get('id') == payload['id']:
            return obj
    raise RuntimeError('timeout')

print('before compaction:', json.loads(settings.read_text())['compaction']['enabled'])
request({'id': 1, 'type': 'get_state'})
request({'id': 2, 'type': 'set_auto_compaction', 'enabled': False})
print('after compaction :', json.loads(settings.read_text())['compaction']['enabled'])
request({'id': 3, 'type': 'set_auto_retry', 'enabled': False})
print('after retry      :', json.loads(settings.read_text())['retry']['enabled'])

proc.terminate()
proc.wait(timeout=5)
PY

Expected:

• before compaction: True
• after compaction : False
• after retry : False

Repro B — prove the shared-config leak

1. Ensure ~/.pi/agent/settings.json has compaction.enabled: true
2. Start any RPC Pi session without setting PI_CODING_AGENT_DIR
3. Send set_auto_compaction { enabled: false }
4. Observe that the real shared ~/.pi/agent/settings.json flips to false

Repro C — actual Symphony symptom

1. Run Symphony with worker.runtime: pi
2. Ensure compaction.enabled: true in the operator config
3. Monitor the file every few seconds
4. Wait for a worker turn
5. Observe it flip back to false

Extra note

This is not limited to compaction. The same leak is present for retry.enabled, because Symphony sends set_auto_retry false the same way and Pi persists it the same way.