Scenario SDK: emit audio refs instead of inline base64

[drewdrewthis]

Summary

Once voice agents land in the Scenario SDK (#355 against #350), audio bytes are currently emitted inline as base64 inside input_audio.data in trace events. Switch to: upload the audio to LangWatch's audio-asset endpoint (presigned PUT) and emit a stable audio_ref content part in the trace event instead.

This implements the SDK-side of the contract defined in langwatch/langwatch#3964.

Reference shape to emit

Replace:

{
  "type": "input_audio",
  "input_audio": {"data": "<base64 wav>", "format": "wav"}
}

with:

{
  "type": "audio_ref",
  "key": "audio/<projectId>/<traceId>/<uuid>.wav",
  "contentType": "audio/wav",
  "durationMs": 4320,
  "format": "wav"
}

Acceptance Criteria

1. When a voice scenario produces audio (TTS output, captured user simulator audio, recorded responses), Scenario SDK calls the LangWatch backend at POST /api/audio-assets/presigned-put to mint a URL, PUTs the bytes, and emits the audio_ref content part in the resulting trace event.
2. Behind a flag (default on) so users can revert to inline base64 if LangWatch ingest is unreachable. Flag name something like LANGWATCH_AUDIO_INLINE=true to force-fallback.
3. Both directions: user simulator audio AND agent audio responses both get offloaded.
4. Streaming: if the audio is being captured chunk-by-chunk (Pipecat, LiveKit, OpenAI Realtime), buffer + upload once per logical utterance, not per chunk. One ref per utterance.
5. Failure mode: if presigned PUT fails (network, auth), fall back to inline base64 emission with a warning log.
6. Tests: unit test for the upload path with a mocked LangWatch endpoint; integration test against staging LangWatch if feasible.

Out of scope

• TS SDK parity for the Scenario adapters — that's covered in langwatch/scenario#372 (Voice TS SDK parity) and depends on this issue landing first.
• Changing the internal AudioChunk format (still PCM16 @ 24kHz mono per Voice Agents #370 design lock Add Mintlify documentation #1).

Dependencies

• Blocked on feat(#350): voice agents — first-class voice in scenario.run() #355 (voice agents foundation merging). Until voice is in the SDK, there's nothing to redirect to S3.
• Hard dep on langwatch/langwatch#3964 for the endpoint and ref shape — but can develop against fixtures once #3964's PR description spec is published.

Why this matters

Without ref-based emission, every voice scenario will either (a) bloat trace payloads with megabytes of base64 WAV, or (b) hit payload caps and ship truncated audio. The LangWatch backend already supports audio asset storage (#3964); this issue is the consumer-side change that makes voice scenarios actually usable for replay/debug.

Part of langwatch/langwatch#1727 (audio-storage epic). Related: langwatch/langwatch#3552 (current inline player), langwatch/scenario#370 (voice epic), langwatch/scenario#350 (voice foundation).

[drewdrewthis]

Full root cause + ordered fix (wire-captured + extractor-executed against real prod data)

The "inline base64 audio / 90 MB getSuiteRunData" symptom AND "voice audio doesn't show in the prod app (see+listen)" are the same multi-component bug. Verified end-to-end (SDK wire capture + running the backend extractor on a real captured payload + live prod probes).

The chain (each link evidence-backed)

1. SDK content shape — convertModelMessagesToAguiMessages (180bab4) emits audio as an input_audio array so the backend can externalise it. ✅
2. SDK event-reporter regression — event-reporter.ts processEventForApi() then re-stringified that array (JSON.stringify(content)) before POSTing MESSAGE_SNAPSHOT, silently undoing step 1 on the scenario-events path. Fixed in → PR fix(voice/sdk): stop re-stringifying input_audio array in event-reporter (completes #451 see+listen) #601 (TDD'd; wire-capture shows "content":"[…]" before → "content":[…] after). ✅
3. Backend ingest + externalisation — the see+listen surface is scenario-events → simulation_runs (the conversation panel), not traces. The backend fix is fix(stored-objects): externalise AI-SDK file+audio parts (scenario voice audio leak) langwatch#4493 (2813e87): scenario-events schema now union(MessageSchema, chatMessageSchema) (accepts arrays) + extractInlineMediaFromEvent at ingest → externalises audio to stored-objects → rewrites content to /api/files/<id> refs. Merged to langwatch main. ✅ (code)
4. Extractor handles the SDK's exact shape — ran #4493's visitContentPartAsync (unmodified) against a real captured input_audio payload (522,240 chars): routed to inputAudio() → externalise → {input_audio:{url:"/api/files/…", mimeType:"audio/pcm16"}} + ref. NOT the drop path. ✅ (executed)
5. Projection + render — the simulation foldProjection consumes the already-rewritten event; the /api/files ref is small → under MAX_MESSAGE_CONTENT_BYTES (64 KB) → no truncation → <MediaPart> player renders. ✅ (source)

The ONLY remaining gate: DEPLOY

• fix(stored-objects): externalise AI-SDK file+audio parts (scenario voice audio leak) langwatch#4493 is merged to main but NOT deployed. Live prod /api/scenario-events still 400s array content ("expected string, received array") = pre-#4493 schema.
• Release order (load-bearing): deploy backend #4493 first / together with publishing the SDK changes. Shipping the SDK before the backend makes every TS voice MESSAGE_SNAPSHOT 400 in prod (worse than today's silent inline bloat). Python is unaffected (still stringifies → accepted, but its audio is also not playable — same root, just no 400).

Pieces

• SDK: PR fix(voice/sdk): stop re-stringifying input_audio array in event-reporter (completes #451 see+listen) #601 (event-reporter array passthrough — this issue's direct fix) + PR fix(voice/ts): explicit EL ConvAI turn-commit so scripted next-turn receive re-engages #596 (EL turn-commit + Python parity), both stacked on feat(typescript-sdk): voice agent testing — consolidated clean stack #561 (voice/372-refactor).
• Backend: fix(stored-objects): externalise AI-SDK file+audio parts (scenario voice audio leak) langwatch#4493 (deploy gate).
• Related: voice/otel: trace export ships inline base64 audio in span attributes (separate path from #451) #599 (OTEL trace path ships inline audio on the wire — bandwidth only, backend truncates on store; separate + lower priority).

Post-deploy verification (ready to run)

Re-run one TS scenario → /api/scenario-events returns 201 (not 400) → the simulation message content carries a /api/files/<id> ref (not [truncated: …]) → the conversation panel renders + plays audio. Reproducible via the public API.