feat(spanner): support user-provided OpenTelemetry for client metrics export - #13741
Conversation
af8fd4a to
818893b
Compare
There was a problem hiding this comment.
Code Review
This pull request introduces support for exporting Cloud Spanner client metrics to a caller-owned OpenTelemetry destination using a new MetricsProvider API, allowing users to configure custom pipelines independently of the default Google Cloud Monitoring export. It also updates documentation, adds comprehensive integration tests, and integrates routed endpoint attributes for location-aware routing. The review feedback highlights a bug in the newly added isEmulatorEnabled() method, where a mismatched protocol prefix (e.g., http://) between the configured host and the emulator host can cause the emulator detection to incorrectly return false.
| public boolean isEmulatorEnabled() { | ||
| return getChannelProvider() == null | ||
| && emulatorHost != null | ||
| && getHost() != null | ||
| && getHost().equals(emulatorHost); | ||
| } |
There was a problem hiding this comment.
The isEmulatorEnabled() method checks getHost().equals(emulatorHost). However, when setEmulatorHost is called with a value like "localhost:1234", the builder prepends "http://" to this.host (making it "http://localhost:1234"), but builder.emulatorHost remains "localhost:1234". This causes getHost().equals(emulatorHost) to return false because of the mismatched "http://" prefix.
To make this robust and handle both cases (with or without the protocol prefix), consider normalizing the comparison or checking if getHost() ends with emulatorHost or starts with it after stripping the protocol. Note that a null check on getHost() is unnecessary here as the host would have already been validated.
@InternalApi
public boolean isEmulatorEnabled() {
if (getChannelProvider() != null || emulatorHost == null) {
return false;
}
String normalizedHost = getHost().startsWith("http://") ? getHost().substring(7) : getHost();
String normalizedEmulatorHost = emulatorHost.startsWith("http://") ? emulatorHost.substring(7) : emulatorHost;
return normalizedHost.equals(normalizedEmulatorHost);
}References
- A null check on a parsed URL component (like the host) is unnecessary if other parts of the constructor would have already failed with a malformed URL (e.g., one missing a scheme).
… export Adds the MetricsProvider family (Default/Noop/CustomOpenTelemetry), SpannerOptions.Builder#setClientMetricsProvider, and SpannerMetrics#configureMeterProviderBuilder so client metrics can be recorded on a caller-owned OpenTelemetry in the spanner/client namespace, decoupled from the Cloud Monitoring built-in metrics flag. Also broadens the emulator gate (SpannerOptions#isEmulatorEnabled, enablegRPCMetrics overload) so no client metrics are recorded against the emulator. The emulator gate normalizes the http(s) scheme away before comparing the configured host with the emulator host.
818893b to
0d0ab92
Compare
) Internal Reference: go/built-in-metric-custom-export Java PR: googleapis/google-cloud-java#13741
🤖 I have created a release *beep* *boop* --- <details><summary>1.90.0</summary> ## [1.90.0](v1.89.0...v1.90.0) (2026-08-24) ### Features * **bigquery-jdbc:** implement TypeRegistry and TypeDescriptor ([#13947](#13947)) ([0557e69](0557e69)) * **bigquery:** add QueryResultsFormat and ArrowSerializationOptions configurations ([#13942](#13942)) ([ff03e19](ff03e19)) * **bigquery:** expose `StatementType` and query execution stats on `TableResult` ([#14145](#14145)) ([7d16de8](7d16de8)) * **bigtable:** enable microsecond timestamps in client ([#14057](#14057)) ([57aaf8d](57aaf8d)) * **bigtable:** route single-entry MutateRows through a point-write c… ([#14028](#14028)) ([a403703](a403703)) * **datastore:** add support for request tags ([#13732](#13732)) ([b1f6186](b1f6186)) * **ftp:** onboard a new library ([#14068](#14068)) ([f41b2d9](f41b2d9)) * **gax:** add ResumableUploadCallable and ResumableUploadCallSettings ([#14052](#14052)) ([a5e26e8](a5e26e8)) * **google/cloud/biglake/hive/v1:** onboard a new library ([#14130](#14130)) ([650c839](650c839)) * **google/maps/mapmanagement/v2:** onboard a new library ([#14131](#14131)) ([7d00726](7d00726)) * **spanner:** support user-provided OpenTelemetry for client metrics export ([#13741](#13741)) ([da74dee](da74dee)) * update API sources and regenerate ([#14000](#14000)) ([9337a93](9337a93)) * **workloadidentity:** onboard a new library ([#14060](#14060)) ([ab226ee](ab226ee)) ### Bug Fixes * add documentation for insertall api that there's no default retry ([#13953](#13953)) ([1fdb4f1](1fdb4f1)) * add retry behavior documentation to insertall interface to clarify the behavior ([#14058](#14058)) ([1b8f9e3](1b8f9e3)) * **auth:** fix JSpecify nullability in UserAuthorizer and TokenStore ([#14150](#14150)) ([0d5fac0](0d5fac0)) * **auth:** fix remaining nullability in UserAuthorizer and Builder ([#14158](#14158)) ([a51bb8d](a51bb8d)) * **auth:** refine JSpecify nullability for ServiceAccountCredentials and UserCredentials ([#14159](#14159)) ([a929250](a929250)) * **bigquery-jdbc:** enable ITOpenTelemetryTest ([#13991](#13991)) ([fa6641b](fa6641b)) * **bigquery-jdbc:** pass connection proxy settings to OpenTelemetry exporters ([#14011](#14011)) ([115b9b3](115b9b3)) * **bigquery-jdbc:** session context propagation when session is enabled ([#14161](#14161)) ([1e74dda](1e74dda)) * **bigtable:** remove heartbeat miss logging ([#14054](#14054)) ([ec17637](ec17637)) * **deps:** update dependency com.google.apis:google-api-services-bigquery to v2-rev20260731-2.0.0 ([#14149](#14149)) ([95f6c38](95f6c38)) * **deps:** update dependency com.google.cloud:libraries-bom to v26.86.0 ([#14103](#14103)) ([cf5697e](cf5697e)) * **gax-httpjson:** reduce Conscrypt fallback error to debug level ([#13962](#13962)) ([8236771](8236771)) * **gax-httpjson:** remove unsupported and deprecated PQC named groups ([#14107](#14107)) ([7604971](7604971)) * **gax:** register Conscrypt SSLContext SPI classes for GraalVM reflection ([#14129](#14129)) ([73c0243](73c0243)) * **samples:** align native profile junit and surefire versions with shared config ([#14096](#14096)) ([2b84133](2b84133)) * **spanner:** add closeAsync to ReadContext and make transaction closing non-blocking ([#14076](#14076)) ([671f892](671f892)) * **spanner:** scope server-timing metrics per call and guard interceptor lifecycle callbacks ([#14053](#14053)) ([f35c570](f35c570)) * **storage:** use JsonUtils for StorageObject serialization in resumable writes and read channels ([#13976](#13976)) ([d94922f](d94922f)) ### Performance Improvements * **bigquery-jdbc:** eliminate dry run to resolve statement type ([#14156](#14156)) ([7109ecd](7109ecd)) * **spanner-jdbc:** cache commonly used query parameter names ([#14036](#14036)) ([1eb6aa3](1eb6aa3)) * **spanner-jdbc:** cache JDBC metadata query strings ([#14041](#14041)) ([31c628f](31c628f)) * **spanner-jdbc:** cache positional to named param conversion ([#14034](#14034)) ([30e031b](30e031b)) ### Dependencies * **gax-httpjson:** upgrade conscrypt-openjdk-uber to 2.6.2 ([#14117](#14117)) ([2f5481a](2f5481a)) * Update gRPC to v1.82.3 ([#13997](#13997)) ([a786107](a786107)) * Upgrade gRPC to v1.82.4 ([#14088](#14088)) ([0c482fe](0c482fe)) ### Documentation * **bigquery-jdbc:** add user guide with connection property and custom endpoint reference ([#13878](#13878)) ([2dde172](2dde172)) * **gax:** update LRO troubleshooting documentation link ([#14108](#14108)) ([4c5bbae](4c5bbae)) * **spanner-jdbc:** update connection_properties.md documentation ([#14035](#14035)) ([b576fe8](b576fe8)) * **spanner:** update CHANGELOG.md for releases 6.117.0 through 6.120.0 ([#13970](#13970)) ([2413811](2413811)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Summary
Adds Client Metrics: a new, opt-in feature that exports Spanner's client-side metrics to a caller-provided OpenTelemetry pipeline (OTLP, Prometheus, any exporter), including on Spanner Omni where the existing Cloud Monitoring metrics are unavailable.
Fully opt-in and fully decoupled from the existing built-in (Cloud Monitoring) metrics. Default behavior is unchanged: without a client-metrics provider, nothing new happens, and built-in metrics continue to export to Cloud Monitoring exactly as before.
Two independent features
setBuiltInMetricsEnabledand theSPANNER_DISABLE_BUILTIN_METRICSenvironment variable, exactly as today. Not available on Spanner Omni.setClientMetricsProvider(...)— aCustomOpenTelemetryMetricsProviderturns it on;NoopMetricsProvider(or no provider) turns it off. Works on all instance types, Omni included.The two are decoupled:
setBuiltInMetricsEnabledand the env var affect only the built-in Cloud Monitoring sink and have no effect on client metrics; the client-metrics provider affects only the caller-owned sink and has no effect on built-in metrics. Under the hood these are the same client instruments, exported under the distinctspanner/clientnamespace — the difference is the export path, not the metrics.Emulator handling
Client metrics are not recorded when Spanner is pointed at the emulator. To make that reliable when the emulator is configured programmatically (via
setEmulatorHost(...)) and not only through theSPANNER_EMULATOR_HOSTenvironment variable, this PR broadens emulator detection:SpannerOptions.isEmulatorEnabled()now also recognizes the builder-configured emulator host, and the connection-check error message is generalized to describe both the environment-variable and programmatic configuration paths. This gates both the client and Cloud Monitoring metrics sinks off against the emulator using a single detection predicate. Called out explicitly here for reviewer visibility.Motivation
Client metrics currently export only to Cloud Monitoring, which is unavailable on Spanner Omni. Customers running on Omni (or who standardize on their own observability stack) had no way to receive these metrics. This lets them route the metrics to any OpenTelemetry exporter, independently of the built-in Cloud Monitoring configuration.
API