NUA — Notebook Use Agent

A two-sided system that lets an orchestrator agent (OpenClaw/NemoClaw, running GLM in a sandbox) see and operate a real desktop IDE through a host-side Visual Computer Use Agent powered by H Company Holo3 vision models — with a live, bidirectional A2A channel between the two.

┌─────────────────────────────┐        ┌──────────────────────────────────────┐
│  SANDBOX (OpenClaw/NemoClaw)│        │  HOST (Windows)                      │
│                             │        │                                      │
│  orchestrator agent (GLM)   │  MCP   │  orch/mcp_server  :8766              │
│   ├─ nua-write-cell ────────┼────────┼─► write_cell → .ipynb on disk        │
│   │                         │  HTTP  │                                      │
│   ├─ holo-cua-audit ───┐    │  jobs  │  cua_runner (FastAPI)  :8765         │
│   ├─ nua-gather ───────┼────┼────────┼─► POST /jobs → audit│gather│figures  │
│   ├─ nua-figures ──────┘    │  + SSE │      │        (agent/core loop)      │
│   │    ▲ live events,       │◄───────┼──────┤ Holo3 VLM ── pyautogui        │
│   │      guidance, frames   │        │      │            ── PyQt5 overlay   │
│   └─ pdb-visualize          │        │      └─ CDP bridge :9223 → py3Dmol   │
│                             │        │           (GLViewer in VS Code)      │
└─────────────────────────────┘        │  nua-tui  (terminal console)         │
                                       └──────────────────────────────────────┘

The three agents

Agent	Endpoint	Character	Purpose
audit	POST /jobs {kind:"audit"}	thorough, plan-driven	Open-ended visual inspection: run cells, verify renders, read UI state, report findings. Maintains an explicit plan queue; answer is gated on completing it.
gather	POST /jobs {kind:"gather"}	fast, low-reasoning	Mechanical multi-target image collection. Thinking off, temperature 0, lean prompt; capture sweeps (orbit_sweep, full_page_scroll, focus_and_zoom_levels) are host-scripted — zero model turns per sweep. Returns an artifact manifest, never inline base64.
figures	POST /jobs {kind:"figures"}	heavy, deeply integrated	Publication-quality scientific images to a FigureSpec: runs cells, styles renders through the CDP bridge, exports exact WebGL canvas pixels (pngURI), enforces host-side quality gates (resolution/aspect/blank/background), and requests code changes from the orchestrator mid-run when needed.

Legacy sync endpoints POST /trigger_audit and POST /gather_data remain as blocking wrappers over the job layer.

The A2A channel

Every run is a job with a replayable, seq-numbered event stream:

Route	Purpose
POST /jobs	submit {kind, intent, active_workspace, spec?, targets?} → 202 {job_id}
GET /jobs/{id}/events?since=N	live SSE stream: status, thought, action, frame (with thumbnail), guidance_ack, code_change_request, result (+10s heartbeats)
GET /jobs/{id}?since=N	status + incremental events (polling fallback)
POST /jobs/{id}/guidance	inject orchestrator guidance — drained at the top of every loop iteration
GET /jobs/{id}/snapshot	PNG of what the agent sees right now
GET /jobs/{id}/artifacts/{path}	fetch any saved frame/figure/manifest
POST /jobs/{id}/cancel	cooperative cancellation

So the orchestrator always sees the NUA's datastream, can inspect the live view on demand, and can steer audit/figures runs mid-process — and the figures agent's request_orchestrator pauses for the reply, closing the code-revision loop without burning iterations.

Host setup (Windows)

git clone <this repo> ; cd <repo>
powershell -ExecutionPolicy Bypass -File setup.ps1     # venv + deps + .env
code --remote-debugging-port=9223 <your-workspace>     # enables the viewer bridge
.\start.ps1 -Tui                                       # servers + terminal console

setup.ps1 prompts for your HAI_API_KEY (portal.hcompany.ai); all other settings have working defaults in .env.example.

The terminal console (nua-tui)

start.ps1 -Tui (or nua-tui directly) opens the console where you actually give prompts to the agents:

• status bar — runner / MCP / CDP-bridge health, model, busy state;
• jobs pane — every job with live thoughts, actions, frame notifications, and figure events; select a row to replay its stream;
• settings — model picker (holo3-1-35b-a3b free / holo3-122b-a10b max, persisted to .env) and active-workspace selection;
• prompt bar — type a task; it is composed with workspace context and handed to your orchestrator via the ORCH_CMD template in .env (e.g. ORCH_CMD=nemoclaw nua-sandbox run --prompt {prompt}), or shown for pasting when unset.

The TUI boots any missing server itself, so nua-tui alone is a complete cold start.

Orchestrator setup (NemoClaw sandbox, once)

# 1. Egress presets (host aliases + RCSB APIs)
nemoclaw <sandbox> policy-add --from-file orch/mcp_server/mcp-egress.yaml
nemoclaw <sandbox> policy-add --from-file orch/skills/pdb-visualize/rcsb-egress.yaml

# 2. Sync the shared A2A client into each skill, then install them
bash orch/lib/sync_common.sh
for s in holo-cua-audit nua-gather nua-figures nua-write-cell pdb-visualize; do
  nemoclaw <sandbox> skill install orch/skills/$s
done

# 3. In-session: connect the notebook-writer MCP channel
mcporter config add workspace http://host.openshell.internal:8766/sse

Skills

Skill	Channel	What it does
holo-cua-audit	jobs API	submit + live-stream an audit; guide.py steers mid-run, watch.py re-attaches
nua-gather	jobs API	fast target collection; returns the artifact manifest
nua-figures	jobs API	figure production from a FigureSpec; supervises code_change_requests
nua-write-cell	MCP :8766	the orchestrator writes notebook code directly (no vision)
pdb-visualize	composite	protein name → PDB ID → py3Dmol cell → visual verification

Division of labor (enforced): the orchestrator authors all code (via MCP write_cell); Holo3 agents click, type verbatim, frame, capture, and verify — they never invent code.

How the agent loop stays grounded

One shared loop (cua_runner/agent/core/) drives all three agents:

• [0,1000] coordinate contract with a <view> tag every turn; ViewState owns crop state, phases (idle → box_drawn → focused → ready), hazard regions, and target coverage — transitions fire only on code-verified events.
• Annotated observations — ruler ticks, the model's own last box, and last click composited onto each screenshot (set-of-mark style).
• Plan queue — audit/figures decompose the task into ordered items; answer is rejected while items are outstanding.
• Predict-then-verify — the model declares expect_change; a real screen diff checks it, and failed predictions force a stated reflection.
• Tiered recovery — identical failed calls are auto-rejected; at the stuck threshold the host acts (resets the view, clears hazards); the zoom-gate oscillation breaker auto-fits the camera when box verdicts ping-pong.
• History hygiene — max 3 screenshots in context; older turns are compacted into a deterministic digest.

Development on Linux/macOS (no GUI, no Windows, no API key)

The whole logic layer runs headless — hardware boundaries are guarded imports:

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-dev.txt -e .
.venv/bin/pytest            # 100 tests: loop gates, jobs API/SSE, strategies, figures, TUI

On-Windows smoke tools live in devtools/ (cdp_smoke_test.py for the bridge, live_test.py for the loop).

Repository layout

cua_runner/            host-side NUA runner (FastAPI, port 8765)
  agent/core/          shared loop: context, profile, session, tool handlers, history
  agent/prompts/       all prompt text (common blocks + per-agent)
  agent/               profiles (loop.py, gather_loop.py), schemas, Holo3 client, ViewState
  figures/             figures agent: spec, quality gates, handlers
  gather/              host-scripted capture strategies
  vision/              capture, coordinate resolution, perceptual diff, annotation
  bridge/              CDP client + shim.js (the only hand-written JS)
  overlay/             PyQt5 transparent overlay (border, focus boxes, animations, status pill)
  jobs.py · runner.py · api_jobs.py · main.py · config.py
nua_tui/               the terminal console
orch/                  orchestrator-side: skills, MCP server, shared A2A client
tests/                 headless test suite
devtools/              on-Windows smoke/diagnostic scripts

All tunables live in cua_runner/config.py and can be set via .env or environment variables.