Requests to LLM providers pass through the gateway as plain HTTPS. The dashboard and the action log cannot say which model was called, how many tokens were used, or what it cost, and rules cannot match on model or streaming mode.
Direction: an llm facet family next to http, sql and k8s, populated by provider-aware endpoint parsing of request and response bodies, with token and cost figures surfaced in the dashboard and available for reporting. Rules matching on provider/model/stream is a natural second step.
Prior art, not vetted as the final shape: #305 added the facet family, multi-family endpoints, and anthropic/openrouter endpoint plugins with SSE usage parsing. The OTel GenAI export that landed in #684 already extracts some of this for spans, so the two should share one parser rather than duplicate it.
Design points to settle: whether provider parsing lives in core endpoint plugins or external plugins, how cost tables are sourced and kept current (the gateway already loads a litellm model list), how usage from streamed responses is captured without buffering whole bodies, and how facets from two families coexist on one action. Sequenced after the current maintenance work.
Requests to LLM providers pass through the gateway as plain HTTPS. The dashboard and the action log cannot say which model was called, how many tokens were used, or what it cost, and rules cannot match on model or streaming mode.
Direction: an
llmfacet family next tohttp,sqlandk8s, populated by provider-aware endpoint parsing of request and response bodies, with token and cost figures surfaced in the dashboard and available for reporting. Rules matching on provider/model/stream is a natural second step.Prior art, not vetted as the final shape: #305 added the facet family, multi-family endpoints, and
anthropic/openrouterendpoint plugins with SSE usage parsing. The OTel GenAI export that landed in #684 already extracts some of this for spans, so the two should share one parser rather than duplicate it.Design points to settle: whether provider parsing lives in core endpoint plugins or external plugins, how cost tables are sourced and kept current (the gateway already loads a litellm model list), how usage from streamed responses is captured without buffering whole bodies, and how facets from two families coexist on one action. Sequenced after the current maintenance work.