fix: Auto-route tool call results to messages_history by emincihangeri · Pull Request #1967 · SAP/ai-sdk-js

emincihangeri · 2026-06-25T13:54:45Z

Context

Closes SAP/ai-sdk-js-backlog#471.

What this PR does and why it is needed

hyperspace-insights · 2026-06-25T13:55:02Z

Summary

The following content is AI-generated and provides a summary of the pull request:

Fix: Auto-Route Tool Call Results to `messages_history`

Bug Fix

🐛 Tool call results passed via messages could contain {{?...}} patterns from external systems (e.g., search results, API responses) that would be incorrectly interpreted as prompt template placeholders. This fix automatically routes tool messages — and any messages preceding them — to messages_history, bypassing prompt templating entirely while keeping the assistant+tool message pair intact for tool_call_id validation.

Changes

packages/orchestration/src/util/module-config.ts: Replaced the simple routeMessagesToHistory boolean flag with a splitIndex calculation. Messages up to and including the last tool message are routed to messages_history; remaining messages stay in prompt.template. The messagesHistory build logic is simplified to always spread messages.slice(0, splitIndex) after any existing messagesHistory.
packages/orchestration/src/orchestration-completion-post-request.test.ts: Added a tool message auto-routing test suite covering:
- Routing tool messages from messages to messages_history
- Preserving existing messagesHistory when appending tool messages
- Ensuring non-tool messages are unaffected
sample-code/src/orchestration.ts: Added two new sample functions:
- orchestrationToolResultInMessages — demonstrates auto-routing of tool results containing {{?...}} syntax
- orchestrationToolResultMaskingInMessagesHistory — verifies masking is still applied to auto-routed tool results
sample-code/src/index.ts: Exported the two new sample functions.
tests/e2e-tests/src/orchestration.test.ts: Added E2E tests for both new sample functions, asserting successful responses and that masking intermediates are present when expected.

🔄 Regenerate and Update Summary
✏️ Insert as PR Description (deletes this comment)
🗑️ Delete comment

PR Bot Information

Version: 1.26.11

Summary Prompt: Default Prompt
GithubContextProvider: chore(deps): Bump @langchain/core from 0.3.30 to 0.3.31 #471
Event Trigger: pull_request.opened
Correlation ID: 2d37aad0-30a2-477f-8ec5-3d6b4ea7f76d
Output Template: Default Template
File Content Strategy: Full file content
LLM: anthropic--claude-4.6-sonnet

davidkna-sap

Just had a quick look, at a glance it mostly looks good.

hyperspace-insights

The PR introduces auto-routing of tool-call result messages to messages_history to prevent {{?...}} content in tool responses from being misinterpreted as prompt template placeholders. There are two logic issues: the TemplateRef branch now passes an empty [] as messages into moduleRequest instead of undefined (a behavioural change from before), and the unconditional messagesHistory array construction silently changes behaviour for callers that supply only messagesHistory with no tool messages — messages_history is now always emitted even when neither condition previously triggered. Please also address the two existing reviewer comments about findLastIndex and the messagesHistory spread condition.

PR Bot Information

Version: 1.26.11

Correlation ID: 319e6368-3269-4852-83ae-651353e8e144
File Content Strategy: Full file content
Event Trigger: issue_comment.created
LLM: anthropic--claude-4.6-sonnet

hyperspace-insights · 2026-06-25T15:07:46Z

-    routeMessagesToHistory && request
-      ? { ...request, messages: undefined }
+    splitIndex > 0 && request
+      ? { ...request, messages: messages.slice(splitIndex) }


Bug: When usesTemplateRef is true, splitIndex is set to messages.length, and messages.slice(0, splitIndex) equals the full messages array. This means all messages are moved to messages_history, and moduleRequest.messages becomes an empty array (messages.slice(messages.length) → []). However, buildCompletionModulesConfig will then throw 'Either a prompt template or messages must be defined.' if the template is also empty — the same error that existed before this refactor when using a TemplateRef (the old code set messages: undefined). The real regression is that moduleRequest.messages is now [] instead of undefined, and buildCompletionModulesConfig checks !request?.messages?.length — so an empty array still passes that guard safely. But passing an empty messages array into prompt.template is a behavioral change from before (previously messages was undefined so nothing was spread). Verify this is intentional; if not, use messages: undefined for the TemplateRef branch.

Suggested change

? { ...request, messages: messages.slice(splitIndex) }

? { ...request, messages: messages.slice(splitIndex).length ? messages.slice(splitIndex) : undefined }

: request;

Double-check suggestion before committing. Edit this comment for amendments.

Please provide feedback on the review comment by checking the appropriate box:

🌟 Awesome comment, a human might have missed that.

✅ Helpful comment

🤷 Neutral

❌ This comment is not helpful

hyperspace-insights · 2026-06-25T15:07:46Z

+  const messagesHistory = [
+    ...(request?.messagesHistory || []),
+    ...messages.slice(0, splitIndex)
+  ];


Logic Error: When splitIndex === 0 and request?.messagesHistory is non-empty, messagesHistory will be a non-empty array (just the existing history), but the condition messagesHistory.length will correctly include it. However, the existing comment already flags the related issue at line 415 — the suggestion there restores the pre-refactor behaviour of preserving request.messagesHistory even when no tool-message splitting occurs. Without that fix, a call with only messagesHistory (no messages with tool results) will still include messages_history in the output, which is a change from the old behaviour where messagesHistory was only spread when the routing condition was true. This is a silent breaking change for callers that rely on messages_history not being emitted.

Suggested change

];

const messagesHistory =

splitIndex > 0 || request?.messagesHistory?.length

? [...(request?.messagesHistory || []), ...messages.slice(0, splitIndex)]

: undefined;

Double-check suggestion before committing. Edit this comment for amendments.

Please provide feedback on the review comment by checking the appropriate box:

🌟 Awesome comment, a human might have missed that.

✅ Helpful comment

🤷 Neutral

❌ This comment is not helpful

jerome-benoit · 2026-06-26T07:25:34Z

+  // 1. TemplateRef configs — the template lives remotely, so messages cannot be merged in.
+  // 2. Tool results — their content comes from external systems and may contain {{?...}}
+  //    patterns that are not user-defined placeholders. We route through the last tool
+  //    message so the assistant+tool pair stays together for tool_call_id validation.


You are changing the initial intent and semantic of a content behind the scene. The user is not even aware of it and has no way to control it.
What is the rationale behind such a debatable decision?
How can you guarantee that a LLM will behave the same way if a content is moved from a designed container for an intent to another one with another intent?
The whole point of an AI SDK is to interface with an API behind an structured abstraction and not to play with the specified structure of the abstraction.

Message history and prompt messages are joined and both sent to the LLM. The primary impact of moving the messages is that it changes how orchestration processes the messages.

Message history and prompt messages are joined and both sent to the LLM. The primary impact of moving the messages is that it changes how orchestration processes the messages.

Where in the context window? The chronological ordering is altered or not? What is the diff of the context window sent to the LLM with or without it? The placement of content in the context window matters, really matters.

Again, why the API is not fixed properly instead of forcing its consumer to workaround it? It's a simple boolean flag triggering two different code paths.

Assuming the split is done properly, the LLM context window will not change (for this PR I haven't done in-depth checks if this is the case here yet). If this is not the case we may have to re-evaluate doing this, and explore alternatives such as documentation changes, as the service rejected your proposed changes. I would also be hesistant to add escaping to avoid influencing the model with invisible unicode characters.

Anyway in some cases is already message history after consultation with the service.

and explore alternatives such as documentation changes, as the service rejected your proposed changes.

But I think it should be put again on the table by people working directly on AI tooling at SAP in a better way that could lead to a proper fix.

I would also be hesistant to add escaping to avoid influencing the model with invisible unicode characters.

I agree that inserting invisible unicode characters is an uglier workaround than moving the culprit content to a place without template syntax interpretation if the moving it not altering the chronological content in the context window, or at least at a minimal level.

Co-authored-by: David Knaack <david.knaack@sap.com>

davidkna-sap · 2026-06-29T13:56:12Z

 {
  "compilerOptions": {
    "target": "ES2022",
+    "lib": ["es2023", "dom"],


Suggested change

"lib": ["es2023", "dom"],

"lib": ["es2023"],

this causes an error on weather-mcp-server.ts. should I work on it in this PR?

Co-authored-by: David Knaack <david.knaack@sap.com>

davidkna-sap · 2026-07-02T15:03:03Z

+      const roles = templating!.map(m => m.role);
+      const toolIdx = roles.lastIndexOf('tool');
+      const userIdx = roles.lastIndexOf('user');
+      expect(toolIdx).toBeGreaterThan(-1);
+      expect(userIdx).toBeGreaterThan(toolIdx);


I would prefer something like this (might have gotten names wrong):

const roles = templating!.map(m => m.role); expect(roles).toEqual(['assistant', 'tool', 'user']);

Co-authored-by: David Knaack <david.knaack@sap.com>

davidkna-sap · 2026-07-03T14:33:32Z

+
+      const templating = response.getIntermediateResults().templating;
+      const roles = templating!.map(m => m.role);
+      expect(roles).toEqual(['assistant', 'tool', 'user']);


[req] This test fails, please resove. The prompt.template with the system prompt is not being properly handled in the routing, and also consider disabling autorouting if something like that is provided.

https://github.com/SAP/ai-sdk-js/actions/runs/28664711694

emincihangeri and others added 2 commits June 25, 2026 15:51

fix: Auto-route tool call results to messages_history

e6e7467

Merge branch 'main' into investigate/tool-result-prompt-templating

a78b377

emincihangeri changed the title ~~fix: Auto-route tool call results to messages_history~~ chore: Auto-route tool call results to messages_history Jun 25, 2026

chore: Fix lint

02e73cb

davidkna-sap changed the title ~~chore: Auto-route tool call results to messages_history~~ fix: Auto-route tool call results to messages_history Jun 25, 2026

davidkna-sap reviewed Jun 25, 2026

View reviewed changes

Comment thread packages/orchestration/src/util/module-config.ts Outdated

Comment thread packages/orchestration/src/util/module-config.ts Outdated

hyperspace-insights Bot reviewed Jun 25, 2026

View reviewed changes

hyperspace-insights Bot deleted a comment from davidkna-sap Jun 25, 2026

jerome-benoit reviewed Jun 26, 2026

View reviewed changes

emincihangeri and others added 4 commits June 29, 2026 14:10

Update packages/orchestration/src/util/module-config.ts

0170e97

Co-authored-by: David Knaack <david.knaack@sap.com>

chore: Add changeset

9772061

Merge branch 'main' into investigate/tool-result-prompt-templating

4711464

chore: Enable ES2023 array methods in tsconfig

ba033a7

davidkna-sap reviewed Jun 29, 2026

View reviewed changes

emincihangeri and others added 3 commits June 29, 2026 15:57

Update tsconfig.json

b733f83

Co-authored-by: David Knaack <david.knaack@sap.com>

fix: address review comments and add message order tests

06831ae

fix: Changes from lint

889db2c

InjunPark-sap requested a review from davidkna-sap June 30, 2026 14:39

davidkna-sap reviewed Jul 2, 2026

View reviewed changes

InjunPark-sap and others added 2 commits July 3, 2026 14:35

Apply suggestions from code review

00deb0c

Co-authored-by: David Knaack <david.knaack@sap.com>

apply feedbacks from david

7c91e68

davidkna-sap requested changes Jul 3, 2026

View reviewed changes

	? { ...request, messages: messages.slice(splitIndex) }
	? { ...request, messages: messages.slice(splitIndex).length ? messages.slice(splitIndex) : undefined }
	: request;

-  ];
+  const messagesHistory =
+    splitIndex > 0 || request?.messagesHistory?.length
+      ? [...(request?.messagesHistory || []), ...messages.slice(0, splitIndex)]
+      : undefined;

Uh oh!

Conversation

emincihangeri commented Jun 25, 2026

Context

What this PR does and why it is needed

Uh oh!

hyperspace-insights Bot commented Jun 25, 2026

Summary

Fix: Auto-Route Tool Call Results to messages_history

Bug Fix

Changes

Uh oh!

davidkna-sap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hyperspace-insights Bot left a comment

Choose a reason for hiding this comment

Uh oh!

hyperspace-insights Bot Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

hyperspace-insights Bot Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerome-benoit Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fix: Auto-Route Tool Call Results to `messages_history`

jerome-benoit Jun 26, 2026 •

edited

Loading