feat: standalone embedded terminal — launch and drive agents from the browser - #347
Open
pablodelucca wants to merge 15 commits into
Open
feat: standalone embedded terminal — launch and drive agents from the browser#347pablodelucca wants to merge 15 commits into
pablodelucca wants to merge 15 commits into
Conversation
…sion, security model
Design doc gating the implementation of an embedded terminal in the standalone
SPA. Records the measured evidence behind the node-pty decision:
- node-pty@1.1.0 ships prebuilds for darwin/win32 only — no linux-x64/arm64, so
`npx pixel-agents` on Linux would need python + a C++ toolchain via node-gyp.
- npm >=11.16 gates lifecycle scripts by default, so node-pty's prebuild copy and
its spawn-helper chmod never run: the module imports fine and then throws
`posix_spawnp failed` on first spawn. Availability must therefore be probed by
actually spawning a PTY, not by a successful require().
- @lydell/node-pty ships all six platform/arch targets as prebuilt
optionalDependencies with no install scripts, and works out of the box.
Also records why raw terminal I/O stays out of the AsyncAPI contract (data plane
vs control plane), and why the terminal WS must authenticate even though /ws does
not in standalone (WebSockets are exempt from CORS, and cors({origin:true})
reflects any origin).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dependencies for the standalone embedded terminal, added ahead of the code that
uses them so the lockfile stays consistent at every commit.
@lydell/node-pty is an OPTIONAL dependency, chosen over the official node-pty
after measuring both (docs/design/standalone-terminal.md):
- node-pty@1.1.0 ships prebuilds for darwin-{x64,arm64} and win32-{x64,arm64}
only. There is no linux-x64/linux-arm64 prebuild, so `npx pixel-agents` on the
most common server/dev-container platform would fall back to `node-gyp rebuild`
and require python plus a C++ toolchain.
- npm >=11.16 gates lifecycle scripts by default, so node-pty's prebuild copy and
its spawn-helper chmod don't run at all: it imports fine and then throws
`posix_spawnp failed` on first spawn.
- @lydell/node-pty ships all six platform/arch targets as prebuilt
optionalDependencies with no install scripts (the model esbuild uses), and
works out of the box.
Optional, so a platform without a prebuild still installs cleanly — the terminal
just reports itself unavailable. The loader tries a candidate list, so anyone who
prefers the Microsoft package can install it and have it picked up.
The PTY candidates are esbuild externals: native .node binaries cannot be
bundled, and the runtime require() must survive so its failure can be caught.
xterm is bundled by Vite into the SPA — the standalone server serves every asset
itself, never a CDN.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gives the standalone server the terminal-lifecycle role vscode.Terminal plays for
the extension. `launchAgent` and `closeAgent` were previously dropped on the
floor in standalone (they fell through to `default:` as "IDE-specific"), so the
browser could only observe agents, never start one.
- ptyModule.ts: loads the first working PTY module from a candidate list.
Availability is established by ACTUALLY SPAWNING a throwaway PTY, not by a
successful require(): under npm's install-script gating the module imports fine
and dies on first spawn, so a try/catch around require would report the
terminal as working and break on the user's first keystroke.
- ptySessionManager.ts: spawn/write/resize/dispose keyed by agent id, bounded
256KB scrollback ring so a reconnecting browser replays recent output instead
of seeing a blank terminal, SIGHUP->SIGKILL escalation.
- standaloneAgentLauncher.ts: mirrors the VS Code launch path rather than sharing
a seam with it (that function imports vscode and takes twelve positional deps;
unifying it would rewrite the VS Code lifecycle and collide with the sibling
branches). Both reuse provider.buildLaunchCommand, so the command is shared.
- terminalRoutes.ts: GET /terminal/:agentId (WS) + GET /api/terminal/session.
Security — the terminal route authenticates in BOTH modes, unlike /ws which
skips auth in standalone reasoning that a loopback bind suffices. That reasoning
doesn't hold for a shell: WebSocket connections are exempt from CORS, so any page
the user visits can open a socket to 127.0.0.1, and cors({origin:true}) reflects
any Origin, which would let any site fetch the token. Hence token + same-origin
guard. The token rides in Sec-WebSocket-Protocol, not a query param, because
standalone runs Fastify with logger:true and would write it to the request log on
every connection.
Protocol additions are strictly additive (three ServerMessage variants). The raw
byte stream stays out of the AsyncAPI unions: it's a data plane, and routing it
through the store's broadcast would fan one agent's output out to every client.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The browser half of standalone launch/focus/close parity: a bottom drawer with
one xterm tab per launched agent. Launching opens the drawer; clicking a
character focuses that agent's tab — the standalone counterpart of VS Code's
terminalRef.show(), resolved client-side because focusAgent has nothing to do
server-side (there is no editor panel to raise).
- terminalClient.ts: per-agent reconnecting WebSocket. Deliberately NOT routed
through MessageTransport: that seam carries the AsyncAPI control plane over one
shared socket, while this is a per-agent raw byte stream. Reconnect replays the
server's scrollback, and 4401/4404 are treated as terminal (retrying a
rejected auth is pointless).
- TerminalPane.tsx: one xterm per agent, driven imperatively through refs (the
pattern OfficeCanvas already uses). Hidden panes stay mounted so switching tabs
doesn't drop the buffer or force a reconnect.
- TerminalDrawer.tsx: tab bar with per-agent connection status.
- BottomToolbar: `+ Agent` was hidden in browser mode ("no terminal to interact
with"). It now shows when the server reports a working PTY, and renders
disabled with the reason when it doesn't — silently missing UI reads as a bug,
and a native module that failed to install is otherwise undiscoverable.
Terminal CONTENT uses a mono font (a TUI mis-renders in FS Pixel Sans); the
drawer chrome around it keeps the pixel font. xterm's theme lives in
constants.ts, the one file where the no-inline-colors rule permits literals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two rendering bugs found by driving the real SPA in a browser; both are
invisible to assertions that only check the terminal has text in it.
1. Terminal content rendered in FS Pixel Sans, not monospace. index.css's base
layer applies `* { font-pixel }` to EVERY element, including the span xterm
renders each cell into, which beat the font xterm sets on its container. A
proportional face breaks column alignment for a full-screen TUI. Re-asserted
the mono stack in the components layer (`.xterm *` outranks `*`), fed from the
same constant xterm measures its cell grid with so CSS can't drift from
metrics. Drawer chrome keeps the pixel font.
2. The drawer is full-width along the bottom and buried the BottomToolbar, so
+ Agent / Layout / Settings became unclickable as soon as an agent launched.
The toolbar now lifts by the drawer's height via a --terminal-drawer-h custom
property published by App. It defaults to 0px, so with no drawer the toolbar
is exactly where it was.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…leak (review) The terminal token endpoint and WS route guarded only same-origin by comparing the request's Origin host against its Host header. A DNS-rebound page sends BOTH Origin: http://evil.com AND Host: evil.com (the Host header is the URL hostname the browser derives from the rebound domain), so the two match and the guard passed. Verified end-to-end against dist/cli.js: a rebound request fetched the real terminal token (HTTP 200) and, combined with the standalone /ws (unauthenticated) launchAgent, attached a live shell. Add a Host-header allowlist: when the server is bound to loopback (the default), the Host must name a loopback host (127.0.0.1 / localhost / ::1). A rebound request's Host is always the attacker's domain, so it is now rejected at both /api/terminal/session (403) and /terminal/:agentId (4401), before any PTY attach or scrollback replay. When the operator has deliberately bound off-loopback (a warned, opt-in exposure) the clause is skipped and the auth token remains the guard — no legitimate LAN Host is broken. Centralises the loopback host set in core constants (reused by the CLI's off-loopback warning). Adds pure-function and live-socket regression tests for the rebinding shape, and corrects the design doc, which claimed the same-origin check blunted rebinding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iew) fetchTerminalSession() memoised the request with `sessionPromise ??= fetch(...)` so all tabs share one call. But a REJECTED promise was cached too: a single transient failure (server momentarily unreachable, a reconnect racing a restart) pinned a rejected promise forever, so every subsequent connect() and reconnect awaited the same rejection and the terminal could never recover even after the server returned. Clear the cache in a .catch so only successful fetches are memoised; the next connect()/reconnect re-fetches. Success-path dedupe is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the terminal panel's tab strip with a transparent card bar drawn over the office's right edge. One card per agent — pixel mug shot, close button, activity dot — extracted into AgentCard.tsx and visible at all times, whether the panel is open or closed. - Panel opens to the right of the bar; no chrome beyond the resize handle and a ghost close button floating top-right (bg on hover only). - No toggle button: selecting an agent (card or character) opens the panel; the card click also selects the character in the office and follows it with the camera, mirroring a canvas click. - Cards are the tabs: mug shot identity (CharacterMugShot crops the front-facing sprite head), status dot for idle/working/attention plus drawer-owned disconnected, awaiting-input and seen-activity tracking in useExtensionMessages. - Panel stays mounted while closed (display:none), so xterm buffers and sockets survive toggling instead of tearing down and reconnecting. - Fix reload restore: the standalone webviewReady burst sends layoutLoaded before existingAgents, so the webview now adds restored agents immediately when layout is already ready — characters reappear in their seats with their persisted palettes instead of vanishing while the cards fall back to skin 0. - Pixel-style the xterm scrollbar: transparent track, slim square thumb, accent on hover; nothing paints when the TUI doesn't overflow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/ws carries the AsyncAPI control plane, and in standalone it can now spawn agents (launchAgent runs claude --dangerously-skip-permissions in a caller-chosen cwd). It previously skipped auth in standalone, trusting the 127.0.0.1 bind — but a browser is a local client, so any page the user visits could open ws://127.0.0.1/ws (WebSocket connections bypass CORS) and drive it. That was an unauthenticated process-spawn / RCE surface. Require the bearer token on /ws in BOTH modes, mirroring the terminal socket's posture: - Non-browser clients (VS Code host, MCP bridge, curl) present it as `Authorization: Bearer <token>` — the MCP bridge already sends this unconditionally, so it complies for free. - The browser SPA can't set a WebSocket header, so it rides the token as the second subprotocol value (CONTROL_WS_PROTOCOL), the same mechanism the terminal socket uses; a `?token=` param is avoided so the token never hits the request log. - Plus the same-origin + loopback-Host guard as the terminal socket, as defence in depth against DNS rebinding. The SPA fetches the token from a new same-origin-guarded /api/session endpoint (standalone only) before connecting; a cross-origin or rebound page can't read it. The transport re-fetches on every (re)connect so a restarted server's new token is picked up, and skips the real socket in Vite dev (browserMock has no server to hand out a token). extractTokenFromProtocolHeader gains an expected-protocol parameter so the terminal and control sockets share one parser. Covered by a new wsAuth integration test (no/wrong/valid token across both header and subprotocol paths; the session endpoint's same-origin guard) and verified live against the built CLI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The terminal is on by default; --no-terminal opts out for watch-only setups (e.g. a dashboard bound off-loopback) where exposing a shell surface is unwanted. A disabled PtySessionManager reports unavailable through the existing plumbing -- availability broadcast, session route, launcher -- so the browser shows a disabled + Agent button with the reason and no downstream code needs a special case. The native PTY module is never resolved or probed when disabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…flow Fills the three coverage gaps on this branch: - standaloneAgentLauncher unit tests: launch registration, real buildLaunchCommand shape, refusal paths that must not burn agent ids, PTY-exit cleanup, and the JSONL adoption poll. - clientMessageHandler terminal control plane: terminalAvailability on webviewReady (incl. the --no-terminal reason and VS Code-mode absence), live-session re-announcement ordering, launchAgent/closeAgent routing. - e2e standalone terminal spec: browser + Agent spawns the mock claude in a server PTY, drawer tab + xterm output, keystroke echo round-trip, reload reattach with scrollback replay, close cleanup; and --no-terminal keeps the office watch-only with hook-driven agents still rendering. The PTY-launch test skips on Windows (spawning the .cmd shim without a shell hop is the exact untested risk the design doc records). The standalone e2e helper learns cliArgs + mockClaude options (mock bin on the host's PATH), and the fixture exposes them via test.use and attaches the mock invocation log on teardown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… terminal panel The standalone page still carried Vite's scaffold <title>webview-ui</title>. The terminal panel's floating close button now reads '>' instead of '×' — it collapses the panel toward the right edge rather than destroying anything, and the glyph should say so. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…creen snapshots
Reloading the standalone page glitched the terminal in two ways, both
rooted in reattach:
1. Narrow-column squeeze. TerminalPane mounted one commit before App
flips the drawer open, so xterm's open() measured its cell grid inside
a display:none subtree; the poisoned metrics made the first visible
fit() compute ~10 columns and resize the PTY to match. open() is now
deferred to the first fit that sees real dimensions (xterm buffers
earlier writes), and a next-frame self-check re-fits whenever the grid
stops matching what the container calls for.
2. Garbled overlapping fragments. Reconnects replayed the raw scrollback
byte ring, which is unreplayable for a full-screen TUI: it starts
mid-stream and its cursor movements assume screen state a fresh client
doesn't have — and with unchanged PTY dims no SIGWINCH ever triggered
a repaint. Each PtySession now mirrors its output into a headless
xterm (@xterm/headless) and every attach sends a {type:"replay"} frame
first: an @xterm/addon-serialize snapshot of the current screen plus
the geometry it was laid out at. The client resets in-band (RIS),
resizes to that geometry, writes the snapshot, then re-fits to its
container. Frames racing the snapshot are queued server-side and
flushed after it, so nothing is dropped or doubled in the handoff.
The mirror lives only in the CLI bundle — the VS Code adapter only ever
type-imports the terminal modules.
Unit tests pin the new semantics (snapshot reflects current screen, not
byte history; bounded scrollback; replay-before-live ordering on a real
socket), and the standalone e2e suite still passes end to end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
🔍 Vercel Preview: https://pixel-agents-w54b-9sdqmgjmk-pablodeluccas-projects.vercel.app Allure report (Linux + macOS + Windows): https://pixel-agents-w54b-9sdqmgjmk-pablodeluccas-projects.vercel.app/reports/allure/ |
Windows E2E shard 3 failed on standalone hosts dying at boot (health ECONNREFUSED, server.json timeouts, resets), intermittently. New on this branch: every standalone host boot runs the PTY availability probe, and on Windows that spawns a ConPTY and kills it immediately — the instant kill races the conpty agent's socket connect, a known node-pty crash mode that can take the whole process down. The probe only exists to catch the POSIX-specific spawn-helper chmod failure (npm script gating); ConPTY has no helper binary, so on win32 a successful import is the whole test and the spawn probe is now skipped. Also: - terminal.spec's win32 skip moves to describe level — the standalone fixture launches the mock-PATH host before a test body runs, so the in-body skip fired too late to prevent the doomed host launch. - launchStandalone folds host stdout/stderr into setup-failure errors; a host that dies at boot on CI previously left nothing to diagnose. - prettier on ptySessionManager.test.ts (CI format check; the commit hook's glob misses nested server test files). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
npx pixel-agentsbecomes a full surface: + Agent in the browser spawnsclaudein a server-side PTY, streams it into an xterm.js drawer next to the office, and routes its hooks to the character — launch, watch, type, and close without VS Code.How it works
PtySessionManager(server/src/terminal/): one PTY per agent via@lydell/node-pty(optional dep,node-ptyfallback, availability probed by actually spawning). PTYs outlive the WebSocket, so a reload reattaches to the still-running session.WS /terminal/:agentId, deliberately outside the AsyncAPI contract (raw byte stream, not control-plane state). Control plane gains three additiveServerMessagevariants (terminalAvailability,terminalSessionOpened,terminalSessionClosed).standaloneAgentLauncher.ts): mirrors the VS Code launch path —provider.buildLaunchCommand(), pre-registered JSONL, existing file-watcher/session-router machinery.TerminalDrawerwith one xterm tab per launched agent (character mug-shot cards double as tabs), standalone-only — VS Code keeps its own terminal panel.PtySessionmirrors output into a headless xterm; every attach sends a@xterm/addon-serializesnapshot of the current screen (a raw byte ring garbles a full-screen TUI). Client-side, xtermopen()is deferred until the pane has real dimensions so cell metrics are never measured while hidden.Security
/terminal/:agentIdis arbitrary code execution, so unlike/wsit authenticates in both modes: bearer token via WS subprotocol (not URL — Fastify logsreq.url), same-origin guard (CORS reflects any origin; WS is CORS-exempt), and a loopback-Hostallowlist that blunts DNS rebinding. Seedocs/design/standalone-terminal.md.Opt-out
--no-terminalkeeps the office watch-only: launch button disabled with the reason as tooltip, hook-driven agents still render. Default is on.Testing
--no-terminalwatch-only path. PTY spawn of the.cmdmock shim is untested on Windows (recorded in the design doc).🤖 Generated with Claude Code