Skip to content

Add LangSmith trace normalization - #1

Draft
sarahwooders wants to merge 11 commits into
mainfrom
agent/langsmith-traces
Draft

Add LangSmith trace normalization#1
sarahwooders wants to merge 11 commits into
mainfrom
agent/langsmith-traces

Conversation

@sarahwooders

@sarahwooders sarahwooders commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a langsmith trajectory source for canonical LangSmith Run[] and /runs/query { "runs": [...] } payloads
  • normalize documented LangChain/LangGraph, OpenAI Chat Completions and Responses, Anthropic Messages, and Vercel AI SDK shapes through the generic run decoder
  • route only roots marked ls_integration: "deepagents" or "deepagents-code" through their aggregate LangGraph outputs.messages, ignoring redundant child spans
  • reconstruct raw Anthropic SSE outputs when custom instrumentation stores the transport stream
  • deduplicate repeated history across stable-ID and semantic message representations while preserving identical text from distinct turns
  • pair tool calls/results, serialize bare tool-output objects, and honor ls_message_view_exclude
  • include canonical Deep Agents, Deep Agents Code, OpenAI Responses, and Anthropic fixtures
  • update the bundled Python CLI, Python parity coverage, and documentation

Why

LangSmith stores traces as canonical Run spans, while conversational content remains integration-specific. The generic adapter reconstructs documented message formats from LLM and tool runs.

Deep Agents is different in one useful, explicit way: its root LangGraph run already contains the complete ordered message state. Processing both that aggregate and its accumulated child LLM/tool spans duplicates history. For deepagents and deepagents-code, the adapter now uses the root aggregate as the canonical trajectory. Claude Code, Codex, and all other integration markers deliberately remain on the existing generic path.

Scope

The adapter accepts canonical Run arrays or { "runs": [...] } envelopes. It does not implement filesystem discovery, LangSmith network fetching, single-run aliases, or a Claude Code/Codex-specific aggregate decoder. The local Codex adapter is unchanged.

Validation

  • bun run check — 51 passed, 7 optional Deep Agents tests skipped without the Python extra; typecheck, bundle consistency, and build passed
  • real Deep Agents SDK environment — 10/10 checkpoint tests passed
  • Python wrapper with the Deep Agents extra — 6/6 tests passed
  • canonical deepagents and deepagents-code fixtures normalize without diagnostics and ignore deliberately redundant child spans
  • LangSmith's public production-shaped Deep Agents trace normalized to one user turn, two tool calls, two linked tool results, and a final response with zero diagnostics
  • Claude Code and Codex integration-marker tests prove both stay on the generic path
  • the previously captured Codex-in-LangSmith account trace is byte-for-byte unchanged from the pre-change normalized result
  • runtime and JSON Schema validation passed for all golden fixtures and the public Deep Agents trace
  • npm pack --dry-run
  • git diff --check and git diff --cached --check
  • git diff origin/main -- src/adapters/codex.ts is empty

Raw account trace content is not included in this PR.

@cpacker cpacker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a trace-loss edge case in aggregate-root selection. findDeepAgentsRoots() can select a root carrying ls_message_view_exclude. The decode loop then skips every child because an aggregate root exists, and skips the root itself because of the exclusion marker. A trace with valid child LLM runs consequently emits no events and fails with missing_user_records.

Please exclude such roots from aggregate-root selection or fall back to processing their children, and add a regression test for an excluded Deep Agents root with valid child spans.

This branch also currently conflicts with main across the adapter registry/types/tests and generated Python bundle, so it needs a substantive rebase and full regeneration before another review.

👾 Generated with Letta Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants