[MEAI] Add OpenTelemetry GenAI semantic convention selection

[rogerbarreto]

Motivation and context

Microsoft.Extensions.AI currently emits only the latest GenAI semantic conventions supported by the package. It does not honor OTEL_SEMCONV_STABILITY_OPT_IN, so applications cannot select the v1.36 compatibility representation.

This change adds both the v1.36 compatibility representation and the latest experimental representation. It keeps the existing default when the environment variable is absent, avoiding a breaking telemetry change for current users. Applications can select v1.36 by defining OTEL_SEMCONV_STABILITY_OPT_IN without gen_ai_latest_experimental.

This support also unblocks consistent .NET observability in Microsoft Agent Framework, which uses OpenTelemetryChatClient for agent, chat, and tool execution telemetry.

Fixes #7709

Description

Semantic convention selection

The resolved mode follows this precedence:

1. Explicit API configuration
2. OTEL_SEMCONV_STABILITY_OPT_IN
3. The existing latest experimental default

Configuration	Emitted representation
Variable absent	Latest experimental
List containing gen_ai_latest_experimental	Latest experimental
Variable defined without the token	v1.36 compatibility
Explicit API selection	Explicitly selected mode

The environment value is parsed as a comma-separated list. The resolved mode is fixed for the lifetime of each instrumentation instance and is exposed for wrappers that need to emit related spans consistently.

v1.36 compatibility representation

The compatibility mode restores the telemetry contract that Microsoft.Extensions.AI emitted before #6767:

• Uses gen_ai.system instead of gen_ai.provider.name.
• Emits system, user, assistant, tool, and choice messages as structured events.
• Uses the v1.36 message and function call body formats.
• Omits attributes, metrics, and values introduced after v1.36.

Latest experimental representation

The latest experimental mode preserves the current behavior:

• Uses gen_ai.provider.name.
• Emits input and output messages through gen_ai.input.messages and gen_ai.output.messages.
• Emits separate instructions through gen_ai.system_instructions.
• Continues to support convention updates after v1.36.

Sensitive content

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT continues to control message bodies, system instructions, function arguments, and function results in both modes.

The change does not introduce the Agent Framework specific ENABLE_MESSAGE_EVENTS setting.

Scope

This pull request updates:

• OpenTelemetryChatClient
• FunctionInvokingChatClient and FunctionInvocationProcessor
• OpenTelemetryEmbeddingGenerator
• Shared OpenTelemetry constants, serializers, and metric helpers
• Public API metadata for semantic convention selection

Compatibility

The recommended behavior preserves the telemetry emitted when no new configuration is provided.

Defining OTEL_SEMCONV_STABILITY_OPT_IN without gen_ai_latest_experimental intentionally changes that instrumentation instance to the v1.36 compatibility representation. Consumers should not expect provider fields or message representations from both modes in the same operation.

The v1.36 GenAI conventions have Development status. This pull request describes them as a compatibility representation rather than a stable representation.

API

The new API is experimental pending owner review:

namespace Microsoft.Extensions.AI;

public enum OpenTelemetryGenAISemanticConvention
{
    Version1_36,
    LatestExperimental,
}

public sealed partial class OpenTelemetryChatClient
{
    public OpenTelemetryGenAISemanticConvention SemanticConvention { get; set; }
}

Testing

The implementation is covered by:

• Environment parsing and unrelated category tokens
• Explicit API precedence
• Per-instance stability after environment changes
• Sensitive content enabled and disabled
• Non-streaming and streaming chat
• Success, failure, and cancellation
• Tool success and failure
• Parent invoke_agent and invoke_workflow mode propagation
• Environment fallback when no telemetry client supplies a mode
• Chat and embedding spans and metrics
• v1.36 structured message events
• Multi-message response aggregation into one v1.36 choice
• Mode-specific field presence and absence
• Duplicate emission prevention

The filtered AI solution builds and tests successfully. The final full build reports existing IDE0055 warnings in unchanged lines of FunctionInvokingChatClient.cs.

Checklist

• The API shape has owner approval.
• The behavior for an absent environment variable is documented.
• Both representations have positive and negative tests.
• Public API metadata is updated.
• XML documentation uses Development and compatibility terminology correctly.
• The Microsoft Agent Framework integration has been validated against the produced package.

[Copilot]

Pull request overview

Adds GenAI semantic convention mode selection to Microsoft.Extensions.AI OpenTelemetry instrumentation, allowing consumers to choose between a v1.36 compatibility representation and the latest experimental representation (with precedence: explicit API > OTEL_SEMCONV_STABILITY_OPT_IN > latest experimental default).

Changes:

• Introduces OpenTelemetryGenAISemanticConvention (experimental) and plumbs it through chat, embeddings, and function-invocation telemetry.
• Adds v1.36-compatible event-based message emission and mode-specific attribute/metric differences.
• Expands test coverage for environment-variable parsing, mode stability, and v1.36-specific outputs.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/Libraries/Microsoft.Extensions.AI.Tests/Embeddings/OpenTelemetryEmbeddingGeneratorTests.cs	Adds v1.36 compatibility assertions for embedding span tags and metrics.
test/Libraries/Microsoft.Extensions.AI.Tests/ChatCompletion/OpenTelemetryEnvironmentVariableTests.cs	Adds tests for OTEL_SEMCONV_STABILITY_OPT_IN parsing and precedence/stability behavior.
test/Libraries/Microsoft.Extensions.AI.Tests/ChatCompletion/OpenTelemetryChatClientTests.cs	Adds v1.36 compatibility tests validating event bodies and absence of latest-only fields.
test/Libraries/Microsoft.Extensions.AI.Tests/ChatCompletion/FunctionInvokingChatClientTests.cs	Extends function-invocation tests to validate both semantic convention modes and propagation.
src/Shared/DiagnosticIds/DiagnosticIds.cs	Adds experiment diagnostic ID for the new semantic convention selection API.
src/Libraries/Microsoft.Extensions.AI/TelemetryHelpers.cs	Adds semantic-convention environment parsing and provider-attribute-name selection helpers.
src/Libraries/Microsoft.Extensions.AI/OpenTelemetryGenAISemanticConvention.cs	Introduces new experimental enum representing v1.36 vs latest experimental emission modes.
src/Libraries/Microsoft.Extensions.AI/OpenTelemetryConsts.cs	Adds constants for stability opt-in env var/token and v1.36 event names/keys.
src/Libraries/Microsoft.Extensions.AI/Microsoft.Extensions.AI.json	Updates public API metadata for the new enum and properties (experimental).
src/Libraries/Microsoft.Extensions.AI/Embeddings/OpenTelemetryEmbeddingGeneratorBuilderExtensions.cs	Documents env-var behavior and override precedence for embeddings instrumentation.
src/Libraries/Microsoft.Extensions.AI/Embeddings/OpenTelemetryEmbeddingGenerator.cs	Adds SemanticConvention property and applies mode-specific tag/metric differences.
src/Libraries/Microsoft.Extensions.AI/Common/OtelV136.cs	Adds v1.36 event body serialization models.
src/Libraries/Microsoft.Extensions.AI/Common/OtelMessageSerializer.cs	Adjusts “latest” serialization model references (namespaced under OtelLatest).
src/Libraries/Microsoft.Extensions.AI/Common/OtelLatest.cs	Adds latest-experimental serialization models previously embedded in chat client.
src/Libraries/Microsoft.Extensions.AI/Common/OtelContext.cs	Updates source-gen serialization context to include both latest and v1.36 models.
src/Libraries/Microsoft.Extensions.AI/Common/FunctionInvocationProcessor.cs	Adds semantic-convention awareness to execute_tool span tags and sensitive fields.
src/Libraries/Microsoft.Extensions.AI/ChatCompletion/OpenTelemetryChatClientBuilderExtensions.cs	Documents env-var behavior and override precedence for chat instrumentation.
src/Libraries/Microsoft.Extensions.AI/ChatCompletion/OpenTelemetryChatClient.cs	Adds SemanticConvention selection; emits v1.36 message/choice events vs latest tags.
src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs	Plumbs semantic convention from agent activity, telemetry client, or environment default.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.