feat(spanner): support user-provided OpenTelemetry for client metrics export (#13741)

## Summary
Adds **Client Metrics**: a new, opt-in feature that exports Spanner's
client-side metrics to a caller-provided OpenTelemetry pipeline (OTLP,
Prometheus, any exporter), including on Spanner Omni where the existing
Cloud Monitoring metrics are unavailable.

Fully opt-in and fully decoupled from the existing built-in (Cloud
Monitoring) metrics. Default behavior is unchanged: without a
client-metrics provider, nothing new happens, and built-in metrics
continue to export to Cloud Monitoring exactly as before.

> **Scope note:** This change was split out of a larger PR into two
focused PRs. This PR contains **only** the Client Metrics export
feature. The optional `endpoint` metric attribute for location-aware
routing is a separate follow-up: #13740.

## Two independent features
- **Built-in metrics** (existing, unchanged): export to Google Cloud
Monitoring. Controlled by `setBuiltInMetricsEnabled` and the
`SPANNER_DISABLE_BUILTIN_METRICS` environment variable, exactly as
today. Not available on Spanner Omni.
- **Client metrics** (new): export the client instruments to a
caller-owned OpenTelemetry. Controlled **solely** by
`setClientMetricsProvider(...)` — a `CustomOpenTelemetryMetricsProvider`
turns it on; `NoopMetricsProvider` (or no provider) turns it off. Works
on all instance types, Omni included.

The two are decoupled: `setBuiltInMetricsEnabled` and the env var affect
only the built-in Cloud Monitoring sink and have no effect on client
metrics; the client-metrics provider affects only the caller-owned sink
and has no effect on built-in metrics. Under the hood these are the same
client instruments, exported under the distinct `spanner/client`
namespace — the difference is the export path, not the metrics.

## Emulator handling
Client metrics are not recorded when Spanner is pointed at the emulator.
To make that reliable when the emulator is configured programmatically
(via `setEmulatorHost(...)`) and not only through the
`SPANNER_EMULATOR_HOST` environment variable, this PR broadens emulator
detection: `SpannerOptions.isEmulatorEnabled()` now also recognizes the
builder-configured emulator host, and the connection-check error message
is generalized to describe both the environment-variable and
programmatic configuration paths. This gates both the client and Cloud
Monitoring metrics sinks off against the emulator using a single
detection predicate. Called out explicitly here for reviewer visibility.

## Motivation
Client metrics currently export only to Cloud Monitoring, which is
unavailable on Spanner Omni. Customers running on Omni (or who
standardize on their own observability stack) had no way to receive
these metrics. This lets them route the metrics to any OpenTelemetry
exporter, independently of the built-in Cloud Monitoring configuration.

## API
```java
SdkMeterProviderBuilder meterProviderBuilder = SdkMeterProvider.builder();
SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder);
// ... attach your exporter to meterProviderBuilder ...
OpenTelemetry otel =
    OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();

SpannerOptions options =
    SpannerOptions.newBuilder()
        .setClientMetricsProvider(new CustomOpenTelemetryMetricsProvider(otel))
        .build();
```

Author: rahul2393

java-spanner/.readme-partials.yaml

@@ -53,29 +53,69 @@ custom_content: |
   ## Metrics
 
   Cloud Spanner client supports [client-side metrics](https://cloud.google.com/spanner/docs/view-manage-client-side-metrics) that you can use along with server-side metrics to optimize performance and troubleshoot performance issues if they occur.
-  
-  Client-side metrics are measured from the time a request leaves your application to the time your application receives the response. 
+
+  Client-side metrics are measured from the time a request leaves your application to the time your application receives the response.
   In contrast, server-side metrics are measured from the time Spanner receives a request until the last byte of data is sent to the client.
-  
-  These metrics are enabled by default. You can opt out of using client-side metrics with the following code:
-  
+
+  The default Cloud Monitoring export for these metrics is enabled by default. You can opt out of the default Cloud Monitoring export with the following code:
+
   ```
   SpannerOptions options = SpannerOptions.newBuilder()
     .setBuiltInMetricsEnabled(false)
     .build();
   ```
-  
-  You can also disable these metrics by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`.
-  
-  > Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
+
+  You can also disable the default Cloud Monitoring export by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`. These controls affect only the Cloud Monitoring export. They do not affect a caller-owned client-metrics export configured with `CustomOpenTelemetryMetricsProvider`.
+
+  > Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data to Cloud Monitoring. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
+
+  #### Exporting client metrics to OpenTelemetry
+
+  Client metrics export to a caller-owned OpenTelemetry destination is controlled by a `MetricsProvider`,
+  set with `SpannerOptions.Builder.setClientMetricsProvider(MetricsProvider)`. The available
+  providers are:
+
+  * `DefaultMetricsProvider` (the default): no caller-owned client-metrics destination is configured.
+    The built-in Cloud Monitoring export follows `setBuiltInMetricsEnabled` and the
+    `SPANNER_DISABLE_BUILTIN_METRICS` environment variable. On Spanner Omni, where the Cloud Monitoring
+    export is not available, the default provider results in no client-metrics export.
+  * `NoopMetricsProvider`: caller-owned client metrics are explicitly disabled. The Cloud Monitoring
+    export is controlled separately.
+  * `CustomOpenTelemetryMetricsProvider`: the same Spanner client instruments are additionally recorded
+    on an `OpenTelemetry` instance that you provide. You own the metrics pipeline (readers, exporters
+    and resource). This custom destination is independent of the built-in Cloud Monitoring export on all
+    instance types, including Spanner Omni (`InstanceType.OMNI`), for which Cloud Monitoring export is
+    not available.
+
+  Client metrics are not recorded when the client runs against the Spanner emulator, regardless of the
+  configured `MetricsProvider`. gRPC-layer metrics are recorded on a custom destination only when the
+  provided `OpenTelemetry` instance is an `OpenTelemetrySdk`.
+
+  When using `CustomOpenTelemetryMetricsProvider`, it is recommended to register the Spanner
+  client-metrics views on a dedicated `SdkMeterProviderBuilder` with
+  `SpannerMetrics.configureMeterProviderBuilder(SdkMeterProviderBuilder)` before creating the
+  `OpenTelemetry` instance. The views rename the raw instruments, apply the Spanner latency
+  histogram buckets and restrict the recorded attributes to the supported client-metric labels.
+
+  ```java
+  SdkMeterProviderBuilder meterProviderBuilder =
+      SdkMeterProvider.builder().registerMetricReader(PeriodicMetricReader.create(myExporter));
+  SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder);
+  OpenTelemetry openTelemetry =
+      OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();
+  SpannerOptions options =
+      SpannerOptions.newBuilder()
+          .setClientMetricsProvider(CustomOpenTelemetryMetricsProvider.create(openTelemetry))
+          .build();
+  ```
 
   ## Traces
-  Cloud Spanner client supports OpenTelemetry Traces, which gives insight into the client internals and aids in debugging/troubleshooting production issues. 
+  Cloud Spanner client supports OpenTelemetry Traces, which gives insight into the client internals and aids in debugging/troubleshooting production issues.
 
   By default, the functionality is disabled. You need to add OpenTelemetry dependencies, enable OpenTelemetry traces and must configure the OpenTelemetry with appropriate exporters at the startup of your application.
 
   See [Configure client-side tracing](https://cloud.google.com/spanner/docs/set-up-tracing#configure-client-side-tracing) for more details on configuring traces.
-  
+
   #### OpenTelemetry Dependencies
 
   If you are using Maven, add this to your pom.xml file
@@ -129,7 +169,7 @@ custom_content: |
 
   Spanner spanner = options.getService();
   ```
-  
+
   #### OpenTelemetry SQL Statement Tracing
   The OpenTelemetry traces that are generated by the Java client include any request and transaction
   tags that have been set. The traces can also include the SQL statements that are executed and the
@@ -149,42 +189,42 @@ custom_content: |
   #### OpenTelemetry API Tracing
   You can enable tracing of each API call that the Spanner client executes with the `enableApiTracing`
   option. These traces also include any retry attempts for an API call:
-    
+
   ```
   SpannerOptions options = SpannerOptions.newBuilder()
   .setOpenTelemetry(openTelemetry)
   .setEnableApiTracing(true)
   .build();
   ```
-    
+
   This option can also be enabled by setting the environment variable
   `SPANNER_ENABLE_API_TRACING=true`.
 
   > Note: The attribute keys that are used for additional information about retry attempts and the number of requests might change in a future release.
-  
-  #### End-to-end Tracing 
-              
+
+  #### End-to-end Tracing
+
   In addition to client-side tracing, you can opt in for [end-to-end tracing](https://cloud.google.com/spanner/docs/tracing-overview#end-to-end-side-tracing). End-to-end tracing helps you understand and debug latency issues that are specific to Spanner such as the following:
   * Identify whether the latency is due to network latency between your application and Spanner, or if the latency is occurring within Spanner.
   * Identify the Google Cloud regions that your application requests are being routed through and if there is a cross-region request. A cross-region request usually means higher latencies between your application and Spanner.
-  
+
   ```
   SpannerOptions options = SpannerOptions.newBuilder()
   .setOpenTelemetry(openTelemetry)
   .setEnableEndToEndTracing(true)
   .build();
   ```
-  
+
   Refer to [Configure end-to-end tracing](https://cloud.google.com/spanner/docs/set-up-tracing#configure-end-to-end-tracing) to configure end-to-end tracing and to understand its attributes.
-  
+
   > Note: End-to-end traces can only be exported to [Cloud Trace](https://cloud.google.com/trace/docs).
-  
-  
+
+
   ## Instrument with OpenCensus
 
   > Note: OpenCensus project is deprecated. See [Sunsetting OpenCensus](https://opentelemetry.io/blog/2023/sunsetting-opencensus/).
   We recommend migrating to OpenTelemetry, the successor project.
-  
+
   ## Migrate from OpenCensus to OpenTelemetry
 
   > Using the [OpenTelemetry OpenCensus Bridge](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-opencensus-shim), you can immediately begin exporting your metrics and traces with OpenTelemetry.
@@ -214,4 +254,4 @@ custom_content: |
 
   Update your dashboards and alerts to reflect below changes
   * **Metrics name** : `cloud.google.com/java` prefix has been removed from OpenTelemery metrics and instead has been added as Instrumenation Scope.
-  * **Metrics namespace** : OpenTelmetry exporters uses `workload.googleapis.com` namespace opposed to `custom.googleapis.com` with OpenCensus. 
+  * **Metrics namespace** : OpenTelmetry exporters uses `workload.googleapis.com` namespace opposed to `custom.googleapis.com` with OpenCensus.

java-spanner/README.md

@@ -155,17 +155,57 @@ Cloud Spanner client supports [client-side metrics](https://cloud.google.com/spa
 Client-side metrics are measured from the time a request leaves your application to the time your application receives the response. 
 In contrast, server-side metrics are measured from the time Spanner receives a request until the last byte of data is sent to the client.
 
-These metrics are enabled by default. You can opt out of using client-side metrics with the following code:
+The default Cloud Monitoring export for these metrics is enabled by default. You can opt out of the default Cloud Monitoring export with the following code:
 
 ```
 SpannerOptions options = SpannerOptions.newBuilder()
   .setBuiltInMetricsEnabled(false)
   .build();
 ```
 
-You can also disable these metrics by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`.
+You can also disable the default Cloud Monitoring export by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`. These controls affect only the Cloud Monitoring export. They do not affect a caller-owned client-metrics export configured with `CustomOpenTelemetryMetricsProvider`.
 
-> Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
+> Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data to Cloud Monitoring. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
+
+#### Exporting client metrics to OpenTelemetry
+
+Client metrics export to a caller-owned OpenTelemetry destination is controlled by a `MetricsProvider`,
+set with `SpannerOptions.Builder.setClientMetricsProvider(MetricsProvider)`. The available
+providers are:
+
+* `DefaultMetricsProvider` (the default): no caller-owned client-metrics destination is configured.
+  The built-in Cloud Monitoring export follows `setBuiltInMetricsEnabled` and the
+  `SPANNER_DISABLE_BUILTIN_METRICS` environment variable. On Spanner Omni, where the Cloud Monitoring
+  export is not available, the default provider results in no client-metrics export.
+* `NoopMetricsProvider`: caller-owned client metrics are explicitly disabled. The Cloud Monitoring
+  export is controlled separately.
+* `CustomOpenTelemetryMetricsProvider`: the same Spanner client instruments are additionally recorded
+  on an `OpenTelemetry` instance that you provide. You own the metrics pipeline (readers, exporters
+  and resource). This custom destination is independent of the built-in Cloud Monitoring export on all
+  instance types, including Spanner Omni (`InstanceType.OMNI`), for which Cloud Monitoring export is
+  not available.
+
+Client metrics are not recorded when the client runs against the Spanner emulator, regardless of the
+configured `MetricsProvider`. gRPC-layer metrics are recorded on a custom destination only when the
+provided `OpenTelemetry` instance is an `OpenTelemetrySdk`.
+
+When using `CustomOpenTelemetryMetricsProvider`, it is recommended to register the Spanner
+client-metrics views on a dedicated `SdkMeterProviderBuilder` with
+`SpannerMetrics.configureMeterProviderBuilder(SdkMeterProviderBuilder)` before creating the
+`OpenTelemetry` instance. The views rename the raw instruments, apply the Spanner latency
+histogram buckets and restrict the recorded attributes to the supported client-metric labels.
+
+```java
+SdkMeterProviderBuilder meterProviderBuilder =
+    SdkMeterProvider.builder().registerMetricReader(PeriodicMetricReader.create(myExporter));
+SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder);
+OpenTelemetry openTelemetry =
+    OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();
+SpannerOptions options =
+    SpannerOptions.newBuilder()
+        .setClientMetricsProvider(CustomOpenTelemetryMetricsProvider.create(openTelemetry))
+        .build();
+```
 
 ## Traces
 Cloud Spanner client supports OpenTelemetry Traces, which gives insight into the client internals and aids in debugging/troubleshooting production issues.