Conversation
|
This should be capability-gated and default to false. Please add a This comment is AI-generated by |
|
I tested the transcript behavior against the live APIs. There are a few concrete AnthropicThe generated flags currently cover:
I tested all four. Each accepted and followed a mid-conversation system message, Anthropic's required message order is important.
The canonical order must be: and for tool loops: So the buffering in
Deferred tools and cachingLate Anthropic tools must be declared at request level with
Anthropic also inserts hidden system scaffolding when any deferred tool exists. We need a stable deferred dummy from the first request whenever there are initial active tools, otherwise introducing the first deferred tool later invalidates most of the cached prefix. I tested the intended shape with cache breakpoints on the last initial active tool, top-level system prompt, and conversation content: With the dummy present from request one: Without it, adding the first deferred tool on request two: The dummy itself cannot carry Change the Anthropic path as follows:
The current code declares every tool through Docs: https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages OpenAI Responses and CodexI tested every model currently marked for these mechanisms. Direct OpenAI, both
All eight accepted the transcript-anchored mechanism and called the loaded tool. Codex:
The important behavior is that
Change all three Responses request builders (
The current Kimi Chat CompletionsSystem text worked on all tested Moonshot models:
Dynamic system-message tools worked on:
They failed with Kimi tool messages are additions-only and must be retained in subsequent requests. A tool-bearing system message cannot also carry Change the Kimi path as follows:
Docs: https://platform.kimi.ai/docs/guide/use-dynamic-tool-loading Capability reporting
Add an explicit Required pi-ai testsPlease add/update focused payload tests for:
Unsupported providers should continue deriving current request-level tools by replaying transcript additions/removals. The target design calls for user-role This comment is AI-generated by |
7f45441 to
9a9591e
Compare
|
Example session transcript showing the system messages. Captured from a real run of Entries are one JSON object per line in the file; pretty-printed here. The only edits: long prompt section text is clipped ( Points of interest:
{
"type": "session",
"id": "loadout-demo",
"version": 3,
"cwd": "/Users/mitsuhiko/Development/pi-mono"
}
{
"type": "model_change",
"id": "ce3c555f",
"parentId": null,
"provider": "openai-codex",
"modelId": "gpt-5.6-terra"
}
{
"type": "thinking_level_change",
"id": "13f2de78",
"parentId": "ce3c555f",
"thinkingLevel": "low"
}
{
"type": "message",
"id": "4074d10e",
"parentId": "13f2de78",
"message": {
"role": "system",
"sections": {
"preamble": "You are an expert coding assistant operating inside pi, a coding agent …",
"tools": "<tools>\n- read: Read file contents\n- bash: Execute bash commands (ls, grep, find, etc.)\n …",
"rules": "<rules>\n- Use bash for file operations like ls, rg, find …",
"docs": "<docs>\nPi documentation (read only when the user asks about pi itself, its SDK, extensions …",
"cwd": "<cwd>\n/Users/mitsuhiko/Development/pi-mono\n</cwd>"
},
"toolsAdded": [
{
"name": "read",
"description": "Read the contents of a file. Supports text files and images (jpg, png, gif, webp, bmp). Images are sent as attachments. For text files, output is truncated to 2000 lines or 50KB (whichever is hit first). Use offset/limit for large files. When you need the full file, continue with offset until complete.",
"parameters": {
"type": "object",
"required": [
"path"
],
"properties": {
"path": {
"type": "string",
"description": "Path to the file to read (relative or absolute)"
},
"offset": {
"type": "number",
"description": "Line number to start reading from (1-indexed)"
},
"limit": {
"type": "number",
"description": "Maximum number of lines to read"
}
}
},
"constrainedSampling": {
"type": "json_schema",
"strict": "prefer"
}
},
{
"name": "bash",
"description": "Execute a bash command in the current working directory. Returns stdout and stderr. Output is truncated to last 2000 lines or 50KB (whichever is hit first). If truncated, full output is saved to a temp file. Optionally provide a timeout in seconds.",
"parameters": {
"type": "object",
"required": [
"command"
],
"properties": {
"command": {
"type": "string",
"description": "Shell command to execute"
},
"timeout": {
"type": "number",
"description": "Timeout in seconds (optional, no default timeout)"
}
}
},
"constrainedSampling": {
"type": "json_schema",
"strict": "prefer"
}
}
],
"timestamp": 1789421512079
}
}
{
"type": "message",
"id": "34d47490",
"parentId": "4074d10e",
"message": {
"role": "user",
"content": [
{
"type": "text",
"text": "What time is it in Vienna? Use a tool."
}
],
"timestamp": 1789421512079
}
}
{
"type": "message",
"id": "1dd944de",
"parentId": "34d47490",
"message": {
"role": "assistant",
"content": [
{
"type": "toolCall",
"id": "call_AxIloA…|fc_03c8d2…",
"name": "bash",
"arguments": {
"command": "TZ=Europe/Vienna date '+%A, %B %-d, %Y — %H:%M:%S %Z'",
"timeout": 10
}
}
],
"model": "openai-codex/gpt-5.6-terra",
"timestamp": 1789421512094
}
}
{
"type": "message",
"id": "4ba47352",
"parentId": "1dd944de",
"message": {
"role": "toolResult",
"toolCallId": "call_AxIloA…|fc_03c8d2…",
"toolName": "bash",
"content": [
{
"type": "text",
"text": "Monday, September 14, 2026 — 23:31:59 CEST"
}
],
"timestamp": 1789421519006
}
}
{
"type": "message",
"id": "40c3a48c",
"parentId": "4ba47352",
"message": {
"role": "assistant",
"content": [
{
"type": "text",
"text": "It’s 23:31 CEST in Vienna (Monday, September 14, 2026)."
}
],
"model": "openai-codex/gpt-5.6-terra",
"timestamp": 1789421519008
}
}
{
"type": "message",
"id": "4fef1fa4",
"parentId": "40c3a48c",
"message": {
"role": "system",
"sections": {
"tools": "<tools>\n- read: Read file contents\n- get_time: Look up the current local time for a city\n …",
"rules": "<rules>\n- Use read to examine files instead of cat or sed.\n- Be concise in your responses …",
"addendum": "<addendum>\nTalk like a pirate.\n</addendum>"
},
"toolsAdded": [
{
"name": "get_time",
"description": "Get the current local time for a city.",
"parameters": {
"type": "object",
"required": [
"city"
],
"properties": {
"city": {
"type": "string",
"description": "City name"
}
}
}
}
],
"toolsRemoved": [
{
"name": "bash"
}
],
"timestamp": 1789421525596
}
}
{
"type": "message",
"id": "010bab5d",
"parentId": "4fef1fa4",
"message": {
"role": "user",
"content": [
{
"type": "text",
"text": "And now?"
}
],
"timestamp": 1789421525596
}
}
{
"type": "message",
"id": "44ebe458",
"parentId": "010bab5d",
"message": {
"role": "assistant",
"content": [
{
"type": "toolCall",
"id": "call_A0U3os…|fc_03c8d2…",
"name": "get_time",
"arguments": {
"city": "Vienna"
}
}
],
"model": "openai-codex/gpt-5.6-terra",
"timestamp": 1789421525613
}
}
{
"type": "message",
"id": "63aa5558",
"parentId": "44ebe458",
"message": {
"role": "toolResult",
"toolCallId": "call_A0U3os…|fc_03c8d2…",
"toolName": "get_time",
"content": [
{
"type": "text",
"text": "The time in Vienna is 22:41."
}
],
"timestamp": 1789421527568
}
}
{
"type": "message",
"id": "2c811842",
"parentId": "63aa5558",
"message": {
"role": "assistant",
"content": [
{
"type": "text",
"text": "Arrr, it’s 22:41 in Vienna."
}
],
"model": "openai-codex/gpt-5.6-terra",
"timestamp": 1789421527571
}
}This comment is AI-generated. |
|
AI generated (analysis by Claude, reviewed by the author) Custom providers that hand-roll the payload lose the prompt and toolsThe agent loop now streams Options
|
|
Had a close read of normalize-context and the compaction interaction — the leading/patch model is a nice fit for the skills hot-swap case. Two things I couldn't fully resolve from the diff:
Otherwise the AgentState.systemPrompt removal reads clean, and the sdk.md migration coverage looks solid. |
|
There are still three blocking issues in Before changing this implementation, the referenced Anthropic documentation MUST be read and understood in full, especially its placement, deferred-tool, and cache-prefix rules. Targeted live API probes must also be done for each payload shape below; unit tests that only assert locally generated JSON are not sufficient to establish provider compatibility. 1. System messages create fake tool results
becomes: Providers may reject the duplicate result or consume the synthetic failure. Anthropic explicitly requires the system message to come after the A system message must not close pending tool accounting. The Anthropic adapter already moves it to a valid boundary after the real result. 2. Anthropic late-tool declarations are invalidThe adapter sends every declared tool as active at the top level, then emits Anthropic specifies that every top-level tool is active immediately unless it has The implementation also needs safe fallback for:
Those cases cannot safely use additive native transitions. Send the final current tool inventory at request level and omit inline changes. 3. Anthropic cache scaffolding is introduced too lateEven after late tools are marked deferred, request one may contain no deferred tools. Adding the first deferred tool on request two changes Anthropic's hidden prompt structure and invalidates much of the cached prefix. Anthropic documents that A stable deferred dummy therefore needs to be present from request one whenever there is at least one active initial tool. This specific dummy strategy follows from the live cache probe rather than being prescribed directly by the documentation. The dummy must not have The probes need to cover at minimum:
This comment is AI-generated by |
Normalize top-level prompts and tools into transcript state while preserving the existing shorthand. Lower later system messages per provider and persist them in coding-agent sessions.
Remove addedToolNames propagation and provider-specific hidden tool loading. Dynamic tool changes now send the complete current tool list and system prompt on the next request.
…ranscript Make the transcript the single source of truth for the system prompt and tools. SystemMessage gains opaque, ordered sections that later messages patch by name; replaying every system message yields the current prompt and tools. Providers that accept system messages mid-conversation (gated per model via supportsMidConvoSystemMessages) send them in place; all others fold them into the leading system message instead of lowering them to user reminders. The agent loop declares state.tools changes to the model before each request, and AgentState.systemPrompt becomes a read-only replay of the transcript. The coding agent emits section patches only and drops its parallel prompt state: TranscriptCapabilities, ModelContextState, system_prompt session entries, and the old-session migration checkpoint. Old sessions get a full patch on their first request. Also fixes setActiveTools() inside before_agent_start being undone by the stale options copy, keeps compaction from summarizing system entries, and uses the previous summary for split-turn compactions.
…prompt options The agent loop now treats tool fields on a pending system message as intent and rewrites them to the delta between the committed transcript and the executable tool set, so a pending addition of a non-executable tool no longer survives replay next to its own removal. The coding agent stores the refreshed prompt options after a mid-run tool change, so session.systemPrompt and ctx.getSystemPrompt() match what the provider receives for the rest of the run.
OpenRouter rejects every request carrying a configuration_update system message for anthropic/claude-opus-5 while accepting it for Fable 5.1. Gate the model instead of the provider.
Flag OpenCode, OpenCode Go, GitHub Copilot, DeepSeek V4 Pro, and OpenRouter's OpenAI models after live verification of each transport: - OpenCode/Copilot Responses: in-place developer messages and additional_tools pass through; tool search is unverified there. - OpenCode/Copilot Claude: system text passes through; tool_addition/tool_removal blocks are rejected, so only supportsMidConvoSystemMessages is set. - Kimi K3: OpenCode forwards the tool-bearing system message; Copilot drops it, so Copilot gets text only. - DeepSeek V4 Pro and openai/* on OpenRouter accept system text in place. Accept dotted Opus 4.8 ids in the Anthropic predicate for Copilot, and document why the Anthropic converter defers later system messages until the next assistant message.
…eam entry points Public entry points (Models, compat stream functions, ModelRuntime) keep accepting Context and call normalizeContext once. Providers, API modules, StreamFn, streamProxy, faux, and extension custom providers receive a TranscriptContext; the prompt and tools live in the transcript's system messages. Merge normalize-context.ts and transcript-state.ts into transcript.ts. The replay helpers take any message list, so the agent package no longer needs getTranscriptSystemMessage wrappers. Add createInitialSystemMessage and declarationsEqual as the single places that build the leading system message and compare tool declarations. Drop createUserTurnAppender, the resolveTranscriptTools closure, the faux prompt serialization special cases, and the renderSystemPromptSections wrapper; buildSystemPrompt renders through getSystemMessageText so the session prompt and the replayed transcript share one rendering rule.
…ool flow Anthropic native tool changes now keep the initial tools active, send every later declaration with defer_loading behind a stable deferred placeholder, and keep removed tools declared, so tool additions and removals no longer invalidate the prompt cache (measured: full miss before, full read after). The current tool list is sent instead when there is no initial tool or a tool was redefined under the same name. transformMessages no longer closes pending tool calls on a system message; it is held until the results are in, so it cannot produce a duplicate synthetic tool result.
e478b30 to
9b8f143
Compare
3 participants
This PR makes system prompt text and tool changes part of the transcript rather than silently rewriting its starting conditions. This lets Pi record when instructions changed or tools became available, restore that state after resuming or navigating branches, and preserve cached prompt prefixes where the upstream supports it.
null, and liststoolsAdded/toolsRemoved(a changed tool definition is a removal plus an addition). A forced or custom prompt is the same mechanism: it patchespreambleand removes the other sections. There is no separate "complete prompt" checkpoint outside compaction; replaying the system messages in order always yields the current prompt and tools.collapseSystemMessages(): the replayed prompt and current tools become the leading system message and later system messages are dropped. That is the old behavior, one cache miss per change.The default system prompt now has a fixed introductory paragraph followed by named XML-style sections:
<tools>,<rules>,<docs>, optional<addendum>,<project_context>and<skills>, and<cwd>. Extensions can edit the structured prompt inputs, contribute named sections, and change the tool loadout; the same diff mechanism applies.Upstream behavior
Whether a model receives updates in place is a catalog compat flag (
supportsMidConvoSystemMessages) set only for verified models. Everything else collapses.openai,openai-codex, and the OpenCode Zen/Go and GitHub Copilot proxies): flagged for GPT-5.4, 5.4-mini, 5.4-pro, 5.5, 5.6-*, and GPT-6 Astra where each endpoint offers them. Instruction updates are in-placedevelopermessages. Tool additions useadditional_toolswhere supported, or completed client-side tool-search entries (Codex GPT-5.5); removals fall back to the complete current request-level tool list. Codex keeps the initial prompt in its separateinstructionsfield. Azure Responses has the transport support but no flagged model.openai/*on OpenRouter, and Copilot Kimi K3. Instruction updates are in-placesystem/developermessages. Kimi-style tool-bearing system messages (supportsMidConvoToolAdditions) are enabled for Kimi K3 on Moonshot, Fireworks, and OpenCode; Copilot silently drops them, so it gets text only. Otherwise tools are sent as the current request-level list. Mistral has the transport support but no flagged model.anthropicprovider. Later system messages aresystem-role messages, and tool changes are nativetool_addition/tool_removalblocks under themid-conversation-tool-changes-2026-07-01beta. Every declared tool is still sent at request level, so native transitions do not guarantee an unchanged tool-schema prefix. Later system messages are emitted directly before the next assistant message sotool_resultstays adjacent totool_use. OpenCode and Copilot Claude endpoints forward the system messages but reject the tool blocks, so they are flagged for text only.The old tool-result-based deferred-loading mechanism is removed, including Anthropic/Fireworks deferred tool references. Dynamic activation works through the transcript transitions or the collapse fallback. Cache preservation depends on the endpoint and transition; it is not universal.
Verification
Live-tested with four scenarios (section update, tool addition, tool removal, tool addition directly after a tool result with no user turn) against every flagged transport: Anthropic Opus 5 / Opus 4.8 / Fable 5.1, OpenAI GPT-5.4 / 5.6 Terra / 6 Astra, Codex GPT-5.5 (tool search) and 5.6 Terra (
additional_tools), Moonshot Kimi K3, and the OpenCode, Copilot, DeepSeek, and OpenRouter entries above with their generated flags. Proxies that reject a native form (Anthropic tool blocks on OpenCode/Copilot, Kimi tools on Copilot) were measured and flagged accordingly.Caching: on api.openai.com an in-place
developerupdate keeps the full cached prefix (measured 3245 read on a 3.3k prompt). The ChatGPT Codex backend uses the legacy 2048-token-interval caching and reports a full miss on the request that introduces adeveloperitem; the collapse strategy would miss there too.Sessions and compaction
Session JSONL stores system messages with sections and tool additions/removals; there is no separate prompt state entry. The session version remains 3. Sessions created before this change have no leading system message; the first request declares the current prompt and tools as a later system message, which replays the same way, rather than rewriting history.
Compaction entries gain a
systemMessagecheckpoint holding the replayed prompt sections and tool declarations at the boundary. Rebuilt context starts with that checkpoint, then the summary and the retained non-system entries; pre-compaction system updates are folded into the checkpoint rather than replayed. Later updates remain chronological. Context estimates include system-message text and tool changes.Also included
fix(ai): OpenRouter rejectsconfiguration_updateeffort messages onanthropic/claude-opus-5(every request 400'd onmain) while accepting them on Fable 5.1; the model is now gated there.