feat(a2ui,tools): add whiteboard, render_ui, charts, reactivity, and surface management

[bizzkoot]

Closes #93

This PR is still work in progress, any comment & suggestion is highly appreciated. Until this is stabilised, I will keep this PR open & in draft. Feel free to pull this PR then build it locally to test the features

Summary

This branch delivers a complete visual interaction layer for Seamless Agent, built across eight commits:

1. feat(tools): implement whiteboard tool — freeform canvas for sketching, annotation, and visual review.
2. feat(tools): add render_ui and refactor whiteboard to image-first contract — structured A2UI panels for forms, dashboards, and guided interactions.
3. feat(a2ui,tools): introduce charts, reactivity, surface management tools, and full render_ui audit — chart components, reactive data binding, surface lifecycle tools, and a complete render_ui capability audit.
4. feat(antigravity): add UI tools to MCP CLI and fix zod/v3 bundling (cde6d0e) — registers all new tools (render_ui, update_ui, append_ui, close_ui, list_surfaces) in the Antigravity MCP CLI; fixes MODULE_NOT_FOUND: zod/v3 error by bundling zod fully into the standalone CLI instead of marking it as an external dependency.
5. fix(a2ui,mcp,tools): resolve 14 audit findings + add smooth curves and calc() support (9f7b74b) — addresses all 15 real issues from docs/audit-validation-report-2026-03-14.md including XSS/data-URL security hardening, DoS protection via chart data limits, MCP handler reliability, and API contract fixes; also implements the smooth bezier curve prop for LineChart and calc() support in CSS dimension validation.
6. fix(mcp,a2ui,webview): multi-instance routing, Mermaid rendering, and history replay (e5daaf4) — fixes three runtime bugs discovered after deployment: render_ui always targeted the first opened IDE window instead of the focused one; Mermaid diagrams rendered as raw text due to a missing CSP unsafe-eval directive and agent confusion over component type; clicking a render_ui history entry had no reopen handler and showed a static text detail instead of re-rendering the panel.
7. fix(mcp): resolve rebase conflicts (5ee5440) — minor cleanup to resolve merge conflicts during rebase.
8. fix(storage,tools,webview): complete async refactoring and test infrastructure (ffab38f) — completes async migration of storage layer, lazy-loads vscode module for better testability, adds imageCleanup utility for canvas export path sanitization, fixes fs mocking to support real file operations in temp directories, updates openWhiteboard tool to properly await storage updates, adds filename indexing to prevent export collisions, removes debug logs, fixes type safety, and improves error context. All 451 tests passing.

Together these give the agent two distinct visual modes plus a rich component catalog:

• Whiteboard for freeform sketching, markup, and screenshot annotation.
• render_ui for structured surfaces with forms, charts, layouts, and real-time reactivity.

The branch also tightens the whiteboard contract around image-first outputs, restores seeded starter-canvas support for agent-authored boards, improves debug coverage, and hardens the tool metadata/schema so the tools register and validate correctly.

Why

This work addresses three related product needs:

1. Agents need a freeform visual collaboration surface.
   Whiteboard covers sketching, markup, screenshots, and iterative visual feedback.

2. Agents also need a structured visual surface.
   Not every interaction should be a drawing canvas. Some flows need forms, content blocks, buttons, Mermaid diagrams, and validation. That is what render_ui provides.

3. Structured surfaces need richer data expression.
   Static panels are not enough. Charts communicate quantitative data. Reactive bindings let agents update panels without full re-renders. Surface management tools (append_ui, update_ui, close_ui) let agents build progressive UIs. A full capability audit ensures the LLM schema matches what the renderer actually supports.

The result is a cleaner split of responsibilities:

• open_whiteboard handles visual sketching and annotation.
• render_ui handles structured A2UI interactions.

Branch Benefit

Capability	What this branch adds	Why it matters
Freeform visual collaboration	open_whiteboard panel, canvas persistence, review flow	Users can sketch or annotate directly instead of forcing everything into text
Image-first visual return path	Exported whiteboard images as the main whiteboard result	Better aligned with model image understanding and review feedback
Agent-authored starter boards	initialCanvases, seedElements, fabricState, importImages	Agents can open blank boards, seeded boards, or annotation-ready image boards
Structured interaction surfaces	render_ui tool and A2UI renderer/panel	Agents can present forms, dashboards, and guided layouts
Chart components	BarChart, LineChart, PieChart in A2UI	Agents can communicate quantitative data visually inside structured panels
Reactive data binding	reactivity.ts with JSON Pointer resolution	Agent-driven data updates without full panel re-renders
Surface lifecycle management	append_ui, update_ui, close_ui, list_surfaces tools	Agents can build progressive, multi-step UIs and clean up precisely
Diagram support inside UI	Mermaid rendering in A2UI panels	Diagrammatic explanation can live inside structured UI
Inline whiteboard in conversations	Whiteboard button in ask_user reply box with image attachment	Users can sketch annotations mid-conversation without explicit tool calls
Smart context restoration	ask_user question is restored after whiteboard submission	Better UX: conversation context is maintained without manual navigation after sketching
Renderer capability audit	Table, Badge, HTML, TextField, Select, ProgressBar schema fixes	LLM schema now accurately reflects actual renderer capabilities
A2UI diagnostics and validation	Required-field validation, droppedStyles, enhancement reports	Agent-authored UI becomes more usable and less error-prone
Better branch operability	Debug fixtures, metadata tests, MCP/native wiring fixes	Easier to test locally and more reliable in actual tool invocation
Antigravity / OVSx MCP CLI fix	All new tools registered in MCP CLI; zod fully bundled (no MODULE_NOT_FOUND: zod/v3)	Antigravity extension starts cleanly for all users on OVSx
Security hardening (XSS, DoS)	Data URL XSS blocked; CSS injection blocked; chart data size limits (25K hard limit)	Prevents crafted agent input from compromising the webview or freezing VS Code
MCP handler reliability	try-catch-finally on all old handlers; CancellationToken always disposed	Malformed inputs return structured errors instead of crashing; no resource leaks
droppedStyles in update_ui result	UpdateUIToolResult now surfaces which CSS properties were dropped	Agents can debug style issues without guessing
Smooth LineChart curves	smooth: true renders cubic bezier path instead of polyline	Cleaner visual output for continuous data series
calc() CSS dimension support	calc(100% - 20px) and similar values pass dimension validation	Agents can use responsive relative layouts in A2UI components
Multi-instance IDE routing	CLI reads a UUID-keyed registry; window focus updates lastActive; every tool call routes to the most-recently focused IDE window	render_ui now targets the active window when multiple IDE instances are open
Mermaid rendering resilience	CSP unsafe-eval for Mermaid v11 JISON parsers; CodeBlock[language=mermaid] and Markdown fenced blocks auto-upgrade to MermaidDiagram rendering	Diagrams render correctly regardless of which component the AI agent selects
CodeBlock prop-name fix	Renderer now reads props.code ?? props.content matching the documented schema	CodeBlock content no longer silently drops when the AI uses the schema-specified prop name
render_ui history replay	Clicking a render_ui history entry reopens the A2UIPanel with stored components and dataModel	History entries for render_ui now behave identically to whiteboard history

Tool Choice

Use case	Preferred tool	Result shape
User sketches a diagram from scratch	open_whiteboard	Exported image(s) of the board
User annotates a screenshot/mockup	open_whiteboard with importImages	Annotated exported image(s)
User wants to sketch mid-conversation	Inline whiteboard in ask_user	Attached whiteboard image(s) to response
Agent needs to show a form or checklist	render_ui	Rendered panel + optional user action
Agent needs a structured dashboard or chart	render_ui with BarChart/LineChart/PieChart	Rendered panel
Agent needs a Mermaid diagram in a guided panel	render_ui	Rendered Mermaid inside panel
Agent needs to update a live panel incrementally	append_ui / update_ui	Updated panel without full re-render
Agent needs to clean up a surface	close_ui	Panel dismissed

Tool Call Flows

Visual collaboration flow

sequenceDiagram
    participant A as Agent
    participant T as open_whiteboard
    participant U as User

    A->>T: Open board
    Note over A,T: blankCanvas or initialCanvases or importImages
    T->>U: Show whiteboard
    U->>T: Draw / annotate / review
    T->>A: Return exported whiteboard images
    Note over T,A: approved | recreateWithChanges | cancelled

Loading

Structured interaction flow

sequenceDiagram
    participant A as Agent
    participant T as render_ui
    participant U as User

    A->>T: Render A2UI surface
    Note over A,T: components + optional dataModel + A2UI processing
    T->>U: Show structured panel
    U->>T: View or submit action
    T->>A: Return rendered state and optional userAction

Loading

Progressive surface flow (new)

sequenceDiagram
    participant A as Agent
    participant S as render_ui / append_ui / update_ui

    A->>S: render_ui — initial components
    S->>A: surfaceId
    A->>S: append_ui — add more components
    A->>S: update_ui — push new dataModel values
    A->>S: close_ui — dismiss when done

Loading

Inline whiteboard flow (ask_user integration)

sequenceDiagram
    participant A as Agent
    participant U as User
    participant W as Inline Whiteboard

    A->>U: ask_user with question
    U->>U: Click whiteboard button
    W->>U: Show inline whiteboard panel
    U->>W: Sketch / annotate
    W->>U: Submit whiteboard
    W->>U: Attach exported image(s) to response
    U->>A: Send response with attached whiteboard image(s)

Loading

Capability map

                +----------------------+
                |   Agent wants UI     |
                +----------+-----------+
                           |
          +----------------+----------------+----------------+
          |                                 |                |
          v                                 v                v
   +--------------+                 +----------------+  +-------------+
   | Freeform or  |                 | Structured or  |  | User wants   |
   | annotation?  |                 | guided UI?     |  | quick sketch |
   +------+-------+                 +--------+-------+  +------+------+
          |                                  |                  |
          v                                  v                  v
   open_whiteboard                     render_ui        Inline whiteboard
   (agent tool)                        (agent tool)      (ask_user button)
          |                                  |                  |
          v                                  v                  v
   exported whiteboard images         panel + optional   attached image(s)
                                       action              to response

What Was Added

1. open_whiteboard

This branch delivers a standalone whiteboard tool with:

• native tool and MCP registration,
• whiteboard panel/webview lifecycle,
• Fabric-backed canvas state handling,
• interaction-history persistence,
• review outcomes for approved, recreateWithChanges, and cancelled,
• image-first exported results,
• support for blank boards, seeded boards, and imported annotation images.

Important behavior:

• blankCanvas: true is the explicit blank-board path.
• initialCanvases supports agent-provided starter canvases.
• seedElements provides coordinate-first starter content for simple agent-authored boards.
• fabricState supports advanced reopen/reseed flows.
• importImages supports screenshot/image annotation workflows.

2. render_ui

This branch also adds a new A2UI-based render path with:

• flat component list contract,
• layout/content/form components,
• A2UI renderer and panel,
• Markdown rendering,
• Mermaid rendering,
• required-field validation before submit,
• optional A2UI processing via enableA2UI and a2uiLevel,
• diagnostics and low-risk surface enhancements when enabled.

This makes render_ui the better fit for structured interactions that should not be modeled as freeform drawing.

3. Chart components (new)

Three new chart components rendering via Chart.js inside the A2UI webview:

Component	props
BarChart	data: {label,value}[], title?, color?, horizontal?, showValues?, style?
LineChart	data: {label,value}[], title?, color?, showPoints?, smooth?, style?
PieChart	data: {label,value,color?}[], title?, doughnut?, showLegend?, style?

Default height 300px, overridable with style: { height: "200px" }.

4. Reactive data binding (new)

reactivity.ts introduces JSON Pointer-based data binding for A2UI:

• Props can reference the dataModel with { "path": "/key/subkey" } syntax.
• update_ui pushes new dataModel values to a live surface, triggering a re-render.
• Enables agent-driven live dashboards and step-by-step panel reveals.

5. Surface lifecycle tools (new)

Tool	Purpose
append_ui	Append new components to an existing surface by surfaceId
update_ui	Push new dataModel to a live surface (triggers reactive re-render)
close_ui	Dismiss a surface panel by surfaceId
list_surfaces	List all currently open surfaces with their IDs and titles

6. render_ui capability audit (new)

A complete audit of the LLM-facing schema vs. the renderer was performed. The following gaps were closed:

Component	Gap fixed
Table	Schema updated: columns changed from string[] to Array<{key,label}>, rows changed to data: Array<Record<string,string|number>>. Key maps to property name in each row.
Badge	Renderer now emits a2ui-badge-{variant} class; CSS rules added for info, success, warning, danger variants.
TextField	helperText?: string documented — shows a small hint below the input.
Select	placeholder?: string and helperText?: string documented.
ProgressBar	showValue?: boolean documented — controls the XX% percentage display.
HTML	css?: string (scoped CSS alongside HTML) and sandbox?: boolean (iframe isolation) documented.

7. Integration and operability improvements

Beyond the tools themselves, the branch includes:

• updated package metadata for all tools,
• metadata regression coverage for tool schemas,
• whiteboard debug fixtures under resources/debug/whiteboard/,
• fixed .vscodeignore to exclude dev artifacts from the packaged VSIX,
• improved local debug flows for whiteboard and A2UI,
• webview badge logic extracted and independently tested,
• MCP server updated with new tool registrations,
• webview/provider wiring so the new surfaces integrate into the existing extension workflow.

8. Inline whiteboard integration

The branch also adds whiteboard access directly within the ask_user workflow:

• Whiteboard button next to the attachment option in ask_user reply box
• Users can open inline whiteboards to sketch annotations or diagrams without leaving the conversation
• Submitted inline whiteboards are attached to the ask_user response as images
• Stacked button layout (attachment on top, whiteboard below) for cleaner UX
• Aspect-ratio-preserving responsive canvas sizing for correct visual proportions
• Smart context restoration: ask_user question is restored after whiteboard submission

UX behavior note: When the whiteboard opens, the sidebar initially shows the pending requests list (VS Code default focus behavior). However, once the whiteboard is submitted, the ask_user question is automatically restored with the attached whiteboard images, maintaining conversation context without requiring manual navigation.

Key Files

Core tool files:

• src/tools/openWhiteboard.ts
• src/tools/renderUI.ts
• src/tools/appendUI.ts
• src/tools/updateUI.ts
• src/tools/closeUI.ts
• src/tools/listSurfaces.ts
• src/tools/schemas.ts
• src/tools/index.ts

Whiteboard implementation:

• src/webview/whiteboardPanel.ts
• src/webview/whiteboard.ts
• src/whiteboard/seededCanvas.ts
• src/storage/chatHistoryStorage.ts

A2UI implementation:

• src/a2ui/catalog.ts
• src/a2ui/engine.ts
• src/a2ui/panel.ts
• src/a2ui/reactivity.ts
• src/a2ui/renderer.ts
• src/a2ui/webview.ts
• media/a2ui.html
• media/a2ui.css

Inline whiteboard integration:

• media/webview.html - ask_user reply box with whiteboard button
• media/main.css - stacked button layout styling
• media/whiteboard.css - aspect-ratio-preserving canvas sizing
• src/webview/webviewProvider.ts - inline whiteboard message handling and render_ui history replay
• src/webview/uiIntegration.test.ts - inline whiteboard integration tests

Multi-instance routing and history replay:

• src/mcp/apiService.ts - UUID-keyed instance registry, focus tracking, dynamic port/token
• bin/seamless-agent-mcp.js - dynamic registry reads, optional startup args
• src/webview/main.ts - renderUI branch in history click handlers
• src/webview/types.ts - openRenderUIPanel message type

9. Antigravity / OVSx MCP CLI fix (cde6d0e)

Registers all new surface tools in the MCP CLI server and resolves the MODULE_NOT_FOUND: zod/v3 startup crash introduced by @modelcontextprotocol/sdk v1.25.2.

Root cause: The old esbuild config marked zod, zod/v3, zod/v4, zod/v4-mini as external for the CLI bundle. The CLI is a standalone self-contained binary deployed inside the extension directory — there is no node_modules alongside it at runtime, so Node.js cannot resolve these external paths. The SDK internally calls require('zod/v3') through its zod-compat.js module, triggering the crash.

Fix: Changed the CLI bundle's external list to ['vscode'] only. Zod is now fully bundled into dist/seamless-agent-mcp.js (1.3 MB self-contained). Verified: grep -c "require(\"zod/v3\")" on the built bundle returns 0.

10. Security hardening and audit fixes (9f7b74b)

Addresses all 15 confirmed real issues (parallel tri-agent audit). 451 tests pass.

Priority	Issue	Fix
P0	XSS via data: URL bypass in sanitizeHTML	Block data:image/svg+xml, data:text/*, vbscript: in all URL attributes; allow safe image MIME types
P1	CSS value injection in parseDeclarativeStyle	sanitizeCssValue() blocks expression(), url(javascript|vbscript|data:text), @import
P1	Unbounded chart data (DoS risk)	CHART_LIMITS constant; hard limit 25K points returns error element; soft limit 10K adds warning
P2	No try-catch on old MCP handlers	ask_user, plan_review, walkthrough_review, open_whiteboard wrapped in try-catch-finally
P2	CancellationTokenSource leaks	Disposed in finally blocks in all 4 old handlers
P2	droppedStyles missing from update_ui result	Added to UpdateUIToolResult schema; collected and returned in updateUI.ts
P2	globalStorageUri unguarded in whiteboardPanel	Null check added for both submit and cancel code paths
P3	Table columns fail silently	Returns <div class="a2ui-error"> when all columns lack a valid key string
P4	Badge variant open class injection	VALID_BADGE_VARIANTS Set whitelist; invalid → default
P4	disabled prop truthy coercion bug	Strict === true check in all components
P4	BarChart incorrect negative value rendering	Proper minValue/maxValue/range spread calculation
P4	Chart color unvalidated	isValidCssColor() validates hex, rgb, hsl, named colors
P5	listSurfaces unused params	Renamed to _params per TypeScript convention

Also implements two previously deferred features:

• smooth prop — LineChart renders cubic bezier <path> via generateSmoothPath() when smooth: true
• calc() support — SAFE_CALC_RE validates calc() expressions blocking dangerous characters

11. Runtime bug fixes — multi-instance routing, Mermaid, and history replay (e5daaf4)

Three bugs discovered after the initial deployment of the Antigravity extension are resolved in this commit.

Bug 1: render_ui always targeted the first opened IDE window

Root cause: apiService.ts wrote a single {port, token} entry to ~/.antigravity/seamless-agent-state.json, overwriting the previous instance. The CLI process held the initial window's port/token in memory and only re-read the file on ECONNREFUSED. With two windows open, the first window was always the target because it never closed.

Fix:

• Each extension instance generates a UUID and writes to a UUID-keyed registry in the state file: { [instanceId]: { port, token, lastActive, startedAt } }.
• vscode.window.onDidChangeWindowState updates lastActive whenever the window gains focus.
• dispose() removes the instance entry on shutdown.
• registerWithAntigravity() no longer bakes --port/--token into mcp_config.json args; the CLI reads them dynamically from the registry.
• The CLI re-reads the registry before every callExtensionApi() call and routes to the entry with the highest lastActive. Backward-compatible with the old flat {port, token} state file format.

Bug 2: Mermaid diagrams rendered as raw text

Root cause (layer 1 — CSP): media/a2ui.html had script-src 'nonce-{{nonce}}' without 'unsafe-eval'. Mermaid v11 uses JISON-based parsers that call new Function() internally. The CSP violation silently caused mermaid.render() to throw, the catch block opened the <details> fallback, and the raw diagram source became visible as plain text.

Root cause (layer 2 — component choice): The AI agent (Gemini) uses Mermaid as a "code fence" pattern from training, so it calls render_ui with {type:"CodeBlock", language:"mermaid", code:"graph TD..."} instead of {type:"MermaidDiagram", definition:"..."}. The renderer also had a prop-name mismatch: the schema says code but the renderer was reading props.content, so CodeBlock content silently dropped.

Fix:

• media/a2ui.html: 'unsafe-eval' added to script-src CSP.
• renderer.ts, CodeBlock case: reads props.code ?? props.content (fixes prop-name mismatch); auto-upgrades to a2ui-mermaid structure when language === 'mermaid'.
• renderer.ts, Markdown case: post-processes rendered HTML to convert <pre><code class="language-mermaid">…</code></pre> to the proper a2ui-mermaid structure, so ```mermaid fenced blocks inside Markdown also render as diagrams.
• schemas.ts: Markdown and CodeBlock descriptions note the auto-upgrade behavior and direct agents toward MermaidDiagram for standalone diagrams.
• 3 new renderer tests added covering: CodeBlock with code prop, CodeBlock[language=mermaid] auto-upgrade, Markdown fenced block auto-upgrade.

Bug 3: Clicking a render_ui history entry showed static text instead of reopening the panel

Root cause: The history-item click handlers in src/webview/main.ts had explicit branches for plan_review (→ openPlanReviewPanel) and whiteboard (→ openWhiteboardPanel) but no branch for renderUI, so it fell through to selectInteraction. _handleSelectInteraction() only sends a showInteractionDetail message back with a static text summary — it never reopens the A2UIPanel.

Fix:

• src/webview/main.ts: added else if (type === 'renderUI') branch in both the View-button handler and the normal item-click handler, dispatching openRenderUIPanel.
• src/webview/types.ts: added openRenderUIPanel to the FromWebviewMessage union.
• src/webview/webviewProvider.ts: added case 'openRenderUIPanel' and implemented _handleOpenRenderUIPanel() — loads the stored renderUISession (components + dataModel) from chatHistoryStorage and calls A2UIPanel.showSurface(), mirroring the existing _handleOpenWhiteboardPanel() pattern.

12. Rebase conflict resolution (5ee5440)

Minor cleanup to resolve merge conflicts during rebase with upstream/main.

Changes:

• src/mcp/mcpServer.ts: Resolved merge conflicts in MCP server implementation.

13. Async refactoring and test infrastructure (ffab38f)

Completes the async migration of the storage layer and improves test infrastructure across storage, tools, and webview components.

Key changes:

Storage layer (async migration):

• src/storage/chatHistoryStorage.ts: Converted all sync storage operations to async
• src/storage/chatHistoryStorage.test.ts: Updated tests for async operations

Test infrastructure improvements:

• src/logging.ts: Lazy-load vscode module for better testability (tests can now run without vscode available)
• src/webview/whiteboardPanel.test.ts: Fixed fs mock to use real fs for operations in tempRoot, enabling proper export testing

New utilities:

• src/whiteboard/imageCleanup.ts: New utility for canvas export path sanitization
• src/whiteboard/imageCleanup.test.ts: Comprehensive tests for image cleanup including cross-platform compatibility (case-sensitivity)
• src/whiteboard/constants.ts: Added constants for image cleanup

Tool fixes:

• src/tools/openWhiteboard.ts: Updated to properly await storage update promises (fixes dismissal hang issue)
• src/tools/planReview.ts: Updated for async storage operations

Code quality improvements:

• src/webview/whiteboardPanel.ts: Removed debug console.log statements, added index to exported canvas filenames to prevent millisecond collisions, improved type safety (changed canvas type from 'any' to NormalizedWhiteboardCanvasSubmission), improved error context

Test coverage:

• All 451 tests passing
• Added 223 lines of imageCleanup tests covering path traversal, case-sensitivity, and edge cases

Verification

Validated with 451 tests, 0 failures across:

• whiteboard schema and package metadata tests,
• render_ui / A2UI tool, panel, renderer, and webview tests,
• chart rendering tests (BarChart, LineChart, PieChart),
• reactivity and JSON Pointer binding tests,
• append_ui, update_ui, close_ui, list_surfaces tool tests,
• MCP registration coverage touched by all new tools,
• compile / type-check validation (zero TypeScript errors),
• regression fix validation for the nested open_whiteboard.seedElements.items schema,
• security fix regressions (sanitizeHTML, parseDeclarativeStyle, chart limits),
• Antigravity MCP CLI bundling (verified: no external require('zod/v3')),
• Mermaid auto-upgrade regressions (CodeBlock prop fix, language=mermaid upgrade, Markdown fence upgrade),
• render_ui history replay (TypeScript type-check, build, integration flow).

Reviewer Guidance

If you want the fastest review path, read these files first:

1. src/tools/schemas.ts — all tool and component contracts
2. src/tools/openWhiteboard.ts
3. src/tools/renderUI.ts
4. src/tools/appendUI.ts, updateUI.ts, closeUI.ts, listSurfaces.ts
5. src/a2ui/reactivity.ts
6. src/a2ui/engine.ts
7. src/a2ui/panel.ts
8. src/a2ui/renderer.ts
9. src/webview/webviewProvider.ts

📸 Visual Changes & Implementations

Feature	Screenshot
ask_user Reply Panel Includes the new #openWhiteboard button.
Whiteboard Interface
Image Attachment Attached image flowing from #openWhiteboard back to the ask_user reply panel.
A2UI Implementation First implementation of A2UI in seamless-agent utilizing the #renderUI tool call.

[bizzkoot]

@iwangbowen & @jraylan , its my first take for this features. It's not perfect, but at least, it does work (agent -> user -> agent). Feel free to test & comments on this. Thanks!

[iwangbowen]

Thanks for the work here, but I think this implementation may have missed the original intent.

What I had in mind were two specific scenarios:

• LLM-initiated drawing session: The LLM opens a whiteboard tool, the user draws on it, and the result is returned to the LLM as an image — which the LLM can then interpret and respond to, potentially asking the user to revise.

• Inline with askUser: The user manually opens a whiteboard as part of an askUser interaction, completes their drawing, and the output is returned as part of that response.

The current implementation essentially defines a new text-based protocol for visual communication — serializing canvas state into structured JSON for the LLM to interpret. I'd argue that use case is already better served by Mermaid diagrams, which are purpose-built for text-based diagram generation.

More importantly, modern LLMs already have native image understanding. We don't need an intermediate text layer to describe visual content — we should just pass the image directly. That's the cleaner, more natural integration.

[jraylan]

Hey, have you guys seem the A2UI? I think it's the right tool for this feature.

[bizzkoot]

Update : will do another pass for this A2UI implementation in tool call #renderUI soon. To make it closer to the approach in v0.8 but still 'native' for seamless-agent extension.

---

Its updated now with A2UI 'implementation' as 1 tool call, #renderUI.
I'm not an expert on this, so feel free to further enhance it if needed.
To be frank, I'm also not sure the full benefit of this within seamless-agent.
But this is to my lack of experiences, so others may feel its helpful?

The brainstorming for this A2UI tool is done with Opus, and implementation with GPT-5.4.
Hopefully it's the correct one. 🤞

For #openWhiteboard, I also add the whiteboard button in the reply box within #askUser.
It's bulky, but not sure where else better to put it. But I do confirm the image attached does work.

Hope this answer the remarks.

[jraylan]

I primarily use AI to compose the frontend.

For me, getting the AI ​​to create the UI the way I imagined it is the hardest part, so yes, I see potential in this, specially for planing.

[bizzkoot]

I have enhanced both #openWhiteboard & #renderUI . Looking forward to hearing your feedback @jraylan & @iwangbowen .

By the way, I'm not an expert for the UI, so any suggestions is highly appreciated. Thanks!

[iwangbowen]

I think it looks great👍

[jraylan]

Probably left by mistake?

[bizzkoot]

Yeah, sorry for this. Is there any other point to be fix? So I could combine it into 1 commit. Thanks!

[jraylan]

Since a lot of file was created/modified, I'll take a few days to fully review this within my routine tasks. But it looks great.