bug(orchestration): getDeltaContent() returns empty string for Anthropic Claude streaming chunks

[madankumarpichamuthu]

Describe the Bug

When using Anthropic Claude models via the Orchestration API in streaming mode, OrchestrationStreamChunkResponse.getDeltaContent() returns undefined (treated as '') for every chunk, causing handleLLMNewToken to fire with empty strings. No text is ever surfaced to the LangChain streaming caller.

Root Cause

ChatDelta.content is typed as string in the generated schema (packages/orchestration/src/client/api/schema/chat-delta.ts):

export type ChatDelta = {
  content: string;
  // ...
};

getDeltaContent() returns it directly:

getDeltaContent(choiceIndex = 0): string | undefined {
  return this.findChoiceByIndex(choiceIndex)?.delta.content;
}

However, Anthropic Claude models return delta.content as Array<{type: string, text: string}> (content blocks) in streaming chunks — not a plain string. The TypeScript type is wrong, and the method has no handling for the array case. The result is that every streaming chunk from a Claude model is silently treated as an empty token.

Steps to Reproduce

1. Configure OrchestrationClient with a Claude model (e.g. claude-3-5-sonnet)
2. Enable streaming ({ streaming: true } in langchainOptions)
3. Call streamEvents({ version: "v3" }) or iterate _streamResponseChunks
4. Observe that all handleLLMNewToken callbacks fire with content = ""

Expected Behavior

getDeltaContent() should extract and concatenate text from content-block arrays when delta.content is an array:

getDeltaContent(choiceIndex = 0): string | undefined {
  const content = this.findChoiceByIndex(choiceIndex)?.delta.content;
  if (Array.isArray(content)) {
    return content
      .filter(b => b.type === 'text')
      .map(b => b.text ?? '')
      .join('');
  }
  return content;
}

The ChatDelta type should also be widened:

export type ChatDelta = {
  content: string | Array<{ type: string; text?: string; [key: string]: any }>;
  // ...
};

Actual Behavior

Every streaming chunk from Claude fires handleLLMNewToken(""). The full response is only available after stream completion via the non-streaming path.

Confirmed Affected Version

@sap-ai-sdk/langchain 2.11.0 (latest at time of filing)

Workaround

Downstream consumers can subclass OrchestrationClient and override _streamResponseChunks to extract text manually from chunk._data:

const rawContent =
  chunk._data?.intermediate_results?.llm?.choices?.[0]?.delta?.content ??
  chunk._data?.final_result?.choices?.[0]?.delta?.content;

const text = Array.isArray(rawContent)
  ? rawContent.filter(b => b.type === 'text').map(b => b.text ?? '').join('')
  : '';

This is the workaround we currently carry in our application. We'd like to see it fixed natively in the SDK.

[KavithaSiva]

Hi @madankumarpichamuthu,

Thanks for the detailed report!
The SAP AI Orchestration service acts as an OpenAI-compatible proxy — regardless of which underlying model you select (Claude, Llama, etc.), the service normalizes all streaming responses to the OpenAI format before returning them to the client. This is by design.

The content-block array format (Array<{type, text}>) you referenced is specific to the native Claude API (platform.anthropic.com). Our SDK does not call Claude directly; it goes through the Orchestration service which handles the translation for the Langchain orchestration client.

If you are genuinely seeing delta.content returned as an array from the Orchestration endpoint, we'd be very interested — but we'd need a raw network trace (the actual SSE response body from the orchestration service) to investigate further.

[madankumarpichamuthu]

Hi @KavithaSiva,

Thanks for the quick response and the clarification!

We captured a raw trace from our production deployment using SAP AI Core Orchestration with a Claude model. Here is the actual chunk.content shape we observe on every streaming chunk:

{"type":"string","value":"\"Here is a 3-bullet summary...\""}
{"type":"string","value":"\" it one of the longest-standing...\""}
{"type":"string","value":"\" by a group of five former IBM employees...\""}

delta.content is always a plain string through the Orchestration proxy — never an Array<{type,text}>. You are correct that the service normalizes all model responses to OpenAI format before returning them.

We are closing the associated PR #1964 as not applicable. The empty streaming tokens we originally observed must have had a different root cause (we suspect it was the missing { streaming: true } constructor option causing silent fallback to the blocking path, which was a separate fix).

Thank you for the pointer — it saved us from shipping an unnecessary workaround into the SDK.