Rich inline cards for tool results via Chat Output Renderer API

[JehutyPT]

PR: Rich Inline Chat Output Rendering

Summary

Integrate VS Code's proposed Chat Output Renderer API to replace plain JSON tool results with rich, themed inline cards in the chat stream for ask_user and plan_review/walkthrough_review tool results.

Before: Tool results rendered as raw markdown: **User Response:** {"responded":true,"response":"yes","attachments":[]}

After: Styled inline cards with question, response, status badges, collapsible plan content, attachments, and timestamps — all themed to match VS Code's color scheme.

Resolves #62

---

Motivation

• Tool output in the @seamless chat participant was raw JSON — ugly, information-poor, and inconsistent with the rich Agent Console webview
• The Chat Output Renderer API enables custom webview widgets inline in chat, providing a polished display layer without replacing the existing input mechanism
• Cards enhance chat history readability when scrolling back through completed interactions

---

What Changed

New Files (5)

File	Purpose
src/proposed-api.d.ts	Type declarations for the proposed chatOutputRenderer API (ExtendedLanguageModelToolResult2, ChatOutputWebview, ChatOutputRenderer, registerChatOutputRenderer). Remove when API stabilizes into @types/vscode.
src/renderers/index.ts	Central registration function with runtime feature gate — no-op on VS Code Stable
src/renderers/types.ts	AskUserRendererData, PlanReviewRendererData interfaces, encodeRendererData/decodeRendererData helpers, truncatePlan utility (50KB max)
src/renderers/htmlUtils.ts	Shared HTML utilities: CSP meta tag, escapeHtml, theme CSS using VS Code CSS variables, localized formatTimestamp, wrapInDocument
src/renderers/askUserRenderer.ts	ask_user card renderer — question section (with truncation/expand), response section, attachment list, dismissed state, all markdown-rendered via markdown-it
src/renderers/planReviewRenderer.ts	Plan review card renderer — status badge (approved/changes requested/cancelled/acknowledged), mode indicator (review/walkthrough), collapsible markdown plan, revision comments

Modified Files (8)

File	Change
package.json	Added enabledApiProposals: ["chatOutputRenderer"] and two chatOutputRenderers contribution entries
src/extension.ts	Import and call registerChatOutputRenderers(context) in activate(); participant handler skips stream.markdown() when toolResultDetails2 is present
src/tools/index.ts	Attach toolResultDetails2 with MIME-typed renderer data to ask_user, plan_review, and walkthrough_review tool results
src/tools/schemas.ts	Added ASK_USER_RESULT_MIME and PLAN_REVIEW_RESULT_MIME constants
src/localization.ts	Added 16 renderer string getters (rendererUserResponse, rendererStatus, rendererViewPlan, etc.)
package.nls.json	16 new English strings for card labels, statuses, and error messages
package.nls.pt-br.json	16 new Brazilian Portuguese strings
package.nls.pt.json	16 new European Portuguese strings (uses distinct terms: "Utilizador", "Estado:")

---

Technical Design

Architecture

Tool invocation flow:
┌─────────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│  Agent calls     │     │  Agent Console       │     │  Tool returns    │
│  ask_user tool   │────▶│  Webview (INPUT)     │────▶│  result + MIME   │
└─────────────────┘     └─────────────────────┘     └──────────────────┘
                                                             │
                                                             ▼
                         ┌─────────────────────────────────────────────┐
                         │  VS Code matches MIME → Chat Output Renderer │
                         │  Renders inline webview card (DISPLAY)       │
                         └─────────────────────────────────────────────┘

Data Flow

1. Tool returns LanguageModelToolResult with both:
   • LanguageModelTextPart (JSON for LLM consumption — unchanged)
   • toolResultDetails2 (MIME + Uint8Array for renderer)
2. VS Code matches MIME type to registered chatOutputRenderers in package.json
3. Renderer receives ChatOutputWebview + binary data, sets webview.html
4. Rich card appears inline in the chat stream

MIME Types

Tool	MIME Type
ask_user	application/vnd.seamless-agent.ask-user-result
plan_review / walkthrough_review	application/vnd.seamless-agent.plan-review-result

Key Design Decisions

Decision	Rationale
enableScripts: false	All HTML is rendered server-side in the extension host. Cards use <details> for interactivity — no JavaScript needed. Simpler CSP, better security.
markdown-it in extension host	Plan and question content rendered to HTML before webview receives it. Avoids loading a library in the webview. Already a dependency at ^14.1.0.
Inline CSS only	No external CSS files — all styles embedded in the HTML template. Eliminates localResourceRoots complexity.
50KB plan truncation	truncatePlan() caps renderer data size. Full plan remains accessible via Agent Console.
Skip approve_plan	Deprecated tool — not worth renderer investment.
Feature-gated registration	registerChatOutputRenderers() checks typeof vscode.chat?.registerChatOutputRenderer !== 'function' at runtime. No-op on VS Code Stable.
Participant handler conflict fix	@seamless handler skips stream.markdown() when toolResultDetails2 is present, preventing the rendered card from being replaced by markdown.

Card Previews

ask_user — Responded:

┌──────────────────────────────────────────────────────┐
│  💬  Agent: User Response                            │
│──────────────────────────────────────────────────────│
│  Question:                                           │
│  "Should I proceed with the deployment?"             │
│                                                      │
│  ✅ User responded:                                  │
│  "Yes, go ahead"                                     │
│                                                      │
│  📎 Attachments:                                     │
│  🖼️ screenshot.png                                   │
│                                    2 minutes ago     │
└──────────────────────────────────────────────────────┘

Plan Review — Changes Requested:

┌──────────────────────────────────────────────────────┐
│  📋  Review: Refactor auth module                    │
│──────────────────────────────────────────────────────│
│  Status: 🔄 Changes Requested                       │
│  ▶ View Plan (collapsed)                             │
│                                                      │
│  💬 Revisions Required:                              │
│  1. "Step 2: Database migration"                     │
│     → "Add rollback strategy"                        │
│                                    3 minutes ago     │
└──────────────────────────────────────────────────────┘

---

Graceful Degradation

Environment	Behavior
VS Code Insiders 1.109+	Rich inline cards rendered via chatOutputRenderer API
VS Code Stable	Renderer not registered (feature gate). toolResultDetails2 silently ignored by VS Code. Participant handler falls back to stream.markdown(**User Response:** ...)
Older VS Code versions	Same as Stable — text-only fallback, no errors

---

Known Limitations

1. Single-webview limitation: VS Code disposes off-screen renderer webviews for memory optimization. Only the most recent card may be visible when scrolling through chat history. This is a VS Code-level behavior.
2. Proposed API: chatOutputRenderer requires enabledApiProposals — blocks VS Code Marketplace publication. Distribution via VSIX only.
3. No thumbnail generation: Image attachments show file icons, not inline previews. Uses webview.asWebviewUri() pattern without base64 encoding.

---

Dependencies

Package	Purpose	Status
markdown-it ^14.1.0	Server-side markdown rendering in extension host	Already a dependency — no new install

No new dependencies added.

---

Testing Status

Scenario	Status
ask_user with response	✅ Tested
ask_user dismissed	⏳ Pending
ask_user with image attachments	⏳ Pending
ask_user with many attachments	⏳ Pending
Plan review approved	⏳ Pending
Plan review changes requested	⏳ Pending
Walkthrough acknowledged	⏳ Pending
Plan review cancelled	⏳ Pending
Chat history scroll-back	⏳ Pending
VS Code Stable fallback (no proposed API)	✅ Tested
Dark theme	✅ Tested
Light theme	⏳ Pending
High contrast theme	⏳ Pending
Malformed renderer data	⏳ Pending
Agent Console webview regression	⏳ Pending
MCP tool invocations unaffected	⏳ Pending

---

Localization

16 new string keys added to all 3 NLS files:

Key	English	Pt-BR	Pt
renderer.userResponse	User Response	Resposta do Usuário	Resposta do Utilizador
renderer.defaultAgentName	Agent	Agente	Agente
renderer.userDismissed	User dismissed this prompt	Usuário dispensou esta solicitação	Utilizador dispensou esta solicitação
renderer.status	Status:	Status:	Estado:
renderer.changesRequested	Changes Requested	Alterações Solicitadas	Alterações Solicitadas
renderer.viewPlan	View Plan	Ver Plano	Ver Plano
renderer.revisionsRequired	Revisions Required:	Revisões Necessárias:	Revisões Necessárias:
...and 9 more

---

Breaking Changes

None. The existing text-based output path is preserved as a fallback. LLM-facing LanguageModelTextPart JSON is unchanged.

---

File Structure

src/
├── proposed-api.d.ts         # Type declarations (remove when API stabilizes)
└── renderers/
    ├── index.ts              # Central registration, feature gate
    ├── types.ts              # Data interfaces, encode/decode, truncatePlan
    ├── htmlUtils.ts          # CSP, escaping, theme CSS, timestamp formatting
    ├── askUserRenderer.ts    # ask_user card renderer + HTML builder
    └── planReviewRenderer.ts # Plan review card renderer + HTML builder

[iwangbowen]

I was looking for a way to bring our sidebar webview Q&A directly into the Copilot chat. The main blocker right now is that the API can't return a webview and simultaneously wait for user interaction before finalizing the tool call.

[jraylan]

Great job.

Since it uses a proposed API, I'm not sure if I can merge this into the main branch, as it will prevent further fix and features from being published.

Maybe creating another branch and link its build into de README.md...

Any suggestion?

[JehutyPT]

Oh yeah, definitely do not merge this in, for the reason you mentioned. I'm fine just letting this sit for now and revisit in the future once the API stabilizes. Just wanted to get your eyes on this to get an initial impression. Want me to move this back into a draft? Or I could close the PR altogether and just reopen it once it stabilizes

[jraylan]

You may let it open. I'll build it for anyone interested in testing it.