Skip to content

Commit 383daa2

Browse files
rahul2393rahul2393
committed
feat(spanner): support user-provided OpenTelemetry for built-in client metrics and endpoint attribute for location-aware routing
1 parent 2f36a71 commit 383daa2

20 files changed

Lines changed: 1843 additions & 61 deletions

‎java-spanner/.readme-partials.yaml‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,18 +57,63 @@ custom_content: |
5757
Client-side metrics are measured from the time a request leaves your application to the time your application receives the response.
5858
In contrast, server-side metrics are measured from the time Spanner receives a request until the last byte of data is sent to the client.
5959
60-
These metrics are enabled by default. You can opt out of using client-side metrics with the following code:
60+
The default Cloud Monitoring export for these metrics is enabled by default. You can opt out of the default Cloud Monitoring export with the following code:
6161
6262
```
6363
SpannerOptions options = SpannerOptions.newBuilder()
6464
.setBuiltInMetricsEnabled(false)
6565
.build();
6666
```
6767
68-
You can also disable these metrics by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`.
68+
You can also disable the default Cloud Monitoring export by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`. If you configure a `CustomOpenTelemetryMetricsProvider` on a non-Omni client, that custom sink is independent and keeps recording.
6969
7070
> Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
7171
72+
#### Customizing how built-in metrics are collected
73+
74+
How built-in metrics are collected and exported is controlled by a `MetricsProvider`,
75+
set with `SpannerOptions.Builder.setMetricsProvider(MetricsProvider)`. The available
76+
providers are:
77+
78+
* `DefaultMetricsProvider` (the default): built-in metrics are collected and exported to
79+
Google Cloud Monitoring.
80+
* `NoopMetricsProvider`: built-in metrics are disabled entirely. This is equivalent to
81+
`setBuiltInMetricsEnabled(false)` only for the default provider; with a
82+
`CustomOpenTelemetryMetricsProvider` set, `setBuiltInMetricsEnabled(false)` turns off
83+
only the Cloud Monitoring sink while `NoopMetricsProvider` disables everything.
84+
* `CustomOpenTelemetryMetricsProvider`: built-in metrics are additionally recorded on an
85+
`OpenTelemetry` instance that you provide. You own the metrics pipeline (readers,
86+
exporters and resource). On non-Omni clients this custom sink is additive to the default
87+
Cloud Monitoring export: both can be active at once. This is also the way to collect
88+
built-in metrics for Spanner Omni (`InstanceType.OMNI`), for which Cloud Monitoring
89+
export is not available.
90+
91+
On non-Omni clients, `setBuiltInMetricsEnabled(false)` and the
92+
`SPANNER_DISABLE_BUILTIN_METRICS` environment variable drop only the Cloud Monitoring
93+
(GCM) sink; a custom sink set via `CustomOpenTelemetryMetricsProvider` is an independent
94+
opt-in and keeps recording. On Omni clients, where the GCM sink is never active, the
95+
`SPANNER_DISABLE_BUILTIN_METRICS` environment variable additionally silences the custom
96+
sink.
97+
98+
When using `CustomOpenTelemetryMetricsProvider`, it is recommended to register the Spanner
99+
built-in metric views on a dedicated `SdkMeterProviderBuilder` with
100+
`SpannerOptions.registerBuiltInMetricViews(SdkMeterProviderBuilder)` before creating the
101+
`OpenTelemetry` instance. The views rename the raw instruments, apply the Spanner latency
102+
histogram buckets and restrict the recorded attributes to the supported built-in metric
103+
labels.
104+
105+
```java
106+
SdkMeterProviderBuilder meterProviderBuilder =
107+
SdkMeterProvider.builder().registerMetricReader(PeriodicMetricReader.create(myExporter));
108+
SpannerOptions.registerBuiltInMetricViews(meterProviderBuilder);
109+
OpenTelemetry openTelemetry =
110+
OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();
111+
SpannerOptions options =
112+
SpannerOptions.newBuilder()
113+
.setMetricsProvider(CustomOpenTelemetryMetricsProvider.create(openTelemetry))
114+
.build();
115+
```
116+
72117
## Traces
73118
Cloud Spanner client supports OpenTelemetry Traces, which gives insight into the client internals and aids in debugging/troubleshooting production issues.
74119

‎java-spanner/README.md‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,18 +155,63 @@ Cloud Spanner client supports [client-side metrics](https://cloud.google.com/spa
155155
Client-side metrics are measured from the time a request leaves your application to the time your application receives the response.
156156
In contrast, server-side metrics are measured from the time Spanner receives a request until the last byte of data is sent to the client.
157157

158-
These metrics are enabled by default. You can opt out of using client-side metrics with the following code:
158+
The default Cloud Monitoring export for these metrics is enabled by default. You can opt out of the default Cloud Monitoring export with the following code:
159159

160160
```
161161
SpannerOptions options = SpannerOptions.newBuilder()
162162
.setBuiltInMetricsEnabled(false)
163163
.build();
164164
```
165165

166-
You can also disable these metrics by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`.
166+
You can also disable the default Cloud Monitoring export by setting `SPANNER_DISABLE_BUILTIN_METRICS` to `true`. If you configure a `CustomOpenTelemetryMetricsProvider` on a non-Omni client, that custom sink is independent and keeps recording.
167167

168168
> Note: Client-side metrics needs `monitoring.timeSeries.create` IAM permission to export metrics data. Ask your administrator to grant your service account the [Monitoring Metric Writer](https://cloud.google.com/iam/docs/roles-permissions/monitoring#monitoring.metricWriter) (roles/monitoring.metricWriter) IAM role on the project.
169169
170+
#### Customizing how built-in metrics are collected
171+
172+
How built-in metrics are collected and exported is controlled by a `MetricsProvider`,
173+
set with `SpannerOptions.Builder.setMetricsProvider(MetricsProvider)`. The available
174+
providers are:
175+
176+
* `DefaultMetricsProvider` (the default): built-in metrics are collected and exported to
177+
Google Cloud Monitoring.
178+
* `NoopMetricsProvider`: built-in metrics are disabled entirely. This is equivalent to
179+
`setBuiltInMetricsEnabled(false)` only for the default provider; with a
180+
`CustomOpenTelemetryMetricsProvider` set, `setBuiltInMetricsEnabled(false)` turns off
181+
only the Cloud Monitoring sink while `NoopMetricsProvider` disables everything.
182+
* `CustomOpenTelemetryMetricsProvider`: built-in metrics are additionally recorded on an
183+
`OpenTelemetry` instance that you provide. You own the metrics pipeline (readers,
184+
exporters and resource). On non-Omni clients this custom sink is additive to the default
185+
Cloud Monitoring export: both can be active at once. This is also the way to collect
186+
built-in metrics for Spanner Omni (`InstanceType.OMNI`), for which Cloud Monitoring
187+
export is not available.
188+
189+
On non-Omni clients, `setBuiltInMetricsEnabled(false)` and the
190+
`SPANNER_DISABLE_BUILTIN_METRICS` environment variable drop only the Cloud Monitoring
191+
(GCM) sink; a custom sink set via `CustomOpenTelemetryMetricsProvider` is an independent
192+
opt-in and keeps recording. On Omni clients, where the GCM sink is never active, the
193+
`SPANNER_DISABLE_BUILTIN_METRICS` environment variable additionally silences the custom
194+
sink.
195+
196+
When using `CustomOpenTelemetryMetricsProvider`, it is recommended to register the Spanner
197+
built-in metric views on a dedicated `SdkMeterProviderBuilder` with
198+
`SpannerOptions.registerBuiltInMetricViews(SdkMeterProviderBuilder)` before creating the
199+
`OpenTelemetry` instance. The views rename the raw instruments, apply the Spanner latency
200+
histogram buckets and restrict the recorded attributes to the supported built-in metric
201+
labels.
202+
203+
```java
204+
SdkMeterProviderBuilder meterProviderBuilder =
205+
SdkMeterProvider.builder().registerMetricReader(PeriodicMetricReader.create(myExporter));
206+
SpannerOptions.registerBuiltInMetricViews(meterProviderBuilder);
207+
OpenTelemetry openTelemetry =
208+
OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();
209+
SpannerOptions options =
210+
SpannerOptions.newBuilder()
211+
.setMetricsProvider(CustomOpenTelemetryMetricsProvider.create(openTelemetry))
212+
.build();
213+
```
214+
170215
## Traces
171216
Cloud Spanner client supports OpenTelemetry Traces, which gives insight into the client internals and aids in debugging/troubleshooting production issues.
172217

‎java-spanner/google-cloud-spanner/src/main/java/com/google/cloud/spanner/BuiltInMetricsConstant.java‎

Lines changed: 83 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,16 @@
4040
public class BuiltInMetricsConstant {
4141

4242
public static final String METER_NAME = "spanner.googleapis.com/internal/client";
43+
44+
/**
45+
* Instrument namespace used when built-in metrics are recorded on a caller-provided {@link
46+
* io.opentelemetry.api.OpenTelemetry} via {@link CustomOpenTelemetryMetricsProvider}. This is
47+
* deliberately distinct from the reserved {@code spanner.googleapis.com} namespace, so that
48+
* re-exporting a caller-owned pipeline to Cloud Monitoring cannot conflict with the curated
49+
* Spanner metric descriptors.
50+
*/
51+
public static final String CUSTOM_EXPORT_METER_NAME = "spanner/internal/client";
52+
4353
public static final String GAX_METER_NAME = OpenTelemetryMetricsRecorder.GAX_METER_NAME;
4454
public static final String GRPC_GCP_METER_NAME = "grpc-gcp";
4555
static final String SPANNER_METER_NAME = "spanner-java";
@@ -168,6 +178,14 @@ public class BuiltInMetricsConstant {
168178
AttributeKey.stringKey("directpath_used");
169179
public static final AttributeKey<String> REQUEST_ID_KEY =
170180
AttributeKey.stringKey(REQUEST_ID_HEADER_NAME);
181+
182+
/**
183+
* The endpoint an attempt was routed to by the location-aware fastpath. Only produced when the
184+
* experimental location API is enabled (Spanner Omni), and only allowed through the attribute
185+
* filter of the custom-export views; the Cloud Monitoring views never include it.
186+
*/
187+
public static final AttributeKey<String> ENDPOINT_KEY = AttributeKey.stringKey("endpoint");
188+
171189
public static Set<String> ALLOWED_EXEMPLARS_ATTRIBUTES =
172190
new HashSet<>(Arrays.asList(REQUEST_ID_HEADER_NAME));
173191

@@ -208,102 +226,138 @@ public class BuiltInMetricsConstant {
208226

209227
static Map<InstrumentSelector, View> getAllViews() {
210228
ImmutableMap.Builder<InstrumentSelector, View> views = ImmutableMap.builder();
229+
addViews(views, METER_NAME, false);
230+
defineGrpcGcpView(views);
231+
return views.build();
232+
}
233+
234+
/**
235+
* Views for built-in metrics recorded on a caller-provided OpenTelemetry via {@link
236+
* CustomOpenTelemetryMetricsProvider}. The instruments live in the {@link
237+
* #CUSTOM_EXPORT_METER_NAME} namespace, and the attribute filter additionally allows the {@link
238+
* #ENDPOINT_KEY} label. The grpc-gcp DirectPath-fallback views are deliberately excluded: those
239+
* metrics are only recorded on the Cloud Monitoring path, so registering their views on a custom
240+
* sink would be dead configuration.
241+
*/
242+
static Map<InstrumentSelector, View> getCustomExportViews() {
243+
ImmutableMap.Builder<InstrumentSelector, View> views = ImmutableMap.builder();
244+
addViews(views, CUSTOM_EXPORT_METER_NAME, true);
245+
return views.build();
246+
}
247+
248+
private static void addViews(
249+
ImmutableMap.Builder<InstrumentSelector, View> views,
250+
String metricPrefix,
251+
boolean includeEndpointAttribute) {
211252
defineView(
212253
views,
254+
metricPrefix,
213255
BuiltInMetricsConstant.GAX_METER_NAME,
214256
BuiltInMetricsConstant.OPERATION_LATENCY_NAME,
215257
BuiltInMetricsConstant.OPERATION_LATENCIES_NAME,
216258
BuiltInMetricsConstant.AGGREGATION_WITH_MILLIS_HISTOGRAM,
217259
InstrumentType.HISTOGRAM,
218-
"ms");
260+
"ms",
261+
includeEndpointAttribute);
219262
defineView(
220263
views,
264+
metricPrefix,
221265
BuiltInMetricsConstant.GAX_METER_NAME,
222266
BuiltInMetricsConstant.ATTEMPT_LATENCY_NAME,
223267
BuiltInMetricsConstant.ATTEMPT_LATENCIES_NAME,
224268
BuiltInMetricsConstant.AGGREGATION_WITH_MILLIS_HISTOGRAM,
225269
InstrumentType.HISTOGRAM,
226-
"ms");
270+
"ms",
271+
includeEndpointAttribute);
227272
defineView(
228273
views,
274+
metricPrefix,
229275
BuiltInMetricsConstant.GAX_METER_NAME,
230276
BuiltInMetricsConstant.OPERATION_COUNT_NAME,
231277
BuiltInMetricsConstant.OPERATION_COUNT_NAME,
232278
Aggregation.sum(),
233279
InstrumentType.COUNTER,
234-
"1");
280+
"1",
281+
includeEndpointAttribute);
235282
defineView(
236283
views,
284+
metricPrefix,
237285
BuiltInMetricsConstant.GAX_METER_NAME,
238286
BuiltInMetricsConstant.ATTEMPT_COUNT_NAME,
239287
BuiltInMetricsConstant.ATTEMPT_COUNT_NAME,
240288
Aggregation.sum(),
241289
InstrumentType.COUNTER,
242-
"1");
243-
defineSpannerView(views);
244-
defineGRPCView(views);
245-
defineGrpcGcpView(views);
246-
return views.build();
290+
"1",
291+
includeEndpointAttribute);
292+
defineSpannerView(views, includeEndpointAttribute);
293+
defineGRPCView(views, metricPrefix, includeEndpointAttribute);
294+
}
295+
296+
private static Set<String> baseAttributesFilter(boolean includeEndpointAttribute) {
297+
Set<String> attributesFilter =
298+
BuiltInMetricsConstant.COMMON_ATTRIBUTES.stream()
299+
.map(AttributeKey::getKey)
300+
.collect(Collectors.toSet());
301+
if (includeEndpointAttribute) {
302+
attributesFilter.add(ENDPOINT_KEY.getKey());
303+
}
304+
return attributesFilter;
247305
}
248306

249307
private static void defineView(
250308
ImmutableMap.Builder<InstrumentSelector, View> viewMap,
309+
String metricPrefix,
251310
String meterName,
252311
String metricName,
253312
String metricViewName,
254313
Aggregation aggregation,
255314
InstrumentType type,
256-
String unit) {
315+
String unit,
316+
boolean includeEndpointAttribute) {
257317
InstrumentSelector selector =
258318
InstrumentSelector.builder()
259-
.setName(BuiltInMetricsConstant.METER_NAME + '/' + metricName)
319+
.setName(metricPrefix + '/' + metricName)
260320
.setMeterName(meterName)
261321
.setType(type)
262322
.setUnit(unit)
263323
.build();
264-
Set<String> attributesFilter =
265-
BuiltInMetricsConstant.COMMON_ATTRIBUTES.stream()
266-
.map(AttributeKey::getKey)
267-
.collect(Collectors.toSet());
268324
View view =
269325
View.builder()
270-
.setName(BuiltInMetricsConstant.METER_NAME + '/' + metricViewName)
326+
.setName(metricPrefix + '/' + metricViewName)
271327
.setAggregation(aggregation)
272-
.setAttributeFilter(attributesFilter)
328+
.setAttributeFilter(baseAttributesFilter(includeEndpointAttribute))
273329
.build();
274330
viewMap.put(selector, view);
275331
}
276332

277-
private static void defineSpannerView(ImmutableMap.Builder<InstrumentSelector, View> viewMap) {
333+
private static void defineSpannerView(
334+
ImmutableMap.Builder<InstrumentSelector, View> viewMap, boolean includeEndpointAttribute) {
278335
InstrumentSelector selector =
279336
InstrumentSelector.builder()
280337
.setMeterName(BuiltInMetricsConstant.SPANNER_METER_NAME)
281338
.build();
282-
Set<String> attributesFilter =
283-
BuiltInMetricsConstant.COMMON_ATTRIBUTES.stream()
284-
.map(AttributeKey::getKey)
285-
.collect(Collectors.toSet());
286-
View view = View.builder().setAttributeFilter(attributesFilter).build();
339+
View view =
340+
View.builder().setAttributeFilter(baseAttributesFilter(includeEndpointAttribute)).build();
287341
viewMap.put(selector, view);
288342
}
289343

290-
private static void defineGRPCView(ImmutableMap.Builder<InstrumentSelector, View> viewMap) {
344+
private static void defineGRPCView(
345+
ImmutableMap.Builder<InstrumentSelector, View> viewMap,
346+
String metricPrefix,
347+
boolean includeEndpointAttribute) {
291348
for (String metric : BuiltInMetricsConstant.GRPC_METRICS_TO_ENABLE) {
292349
InstrumentSelector selector =
293350
InstrumentSelector.builder()
294351
.setName(metric)
295352
.setMeterName(BuiltInMetricsConstant.GRPC_METER_NAME)
296353
.build();
297-
Set<String> attributesFilter =
298-
BuiltInMetricsConstant.COMMON_ATTRIBUTES.stream()
299-
.map(AttributeKey::getKey)
300-
.collect(Collectors.toSet());
354+
Set<String> attributesFilter = baseAttributesFilter(includeEndpointAttribute);
301355
attributesFilter.addAll(
302356
GRPC_METRIC_ADDITIONAL_ATTRIBUTES.getOrDefault(metric, ImmutableSet.of()));
303357

304358
View view =
305359
View.builder()
306-
.setName(BuiltInMetricsConstant.METER_NAME + '/' + metric.replace(".", "/"))
360+
.setName(metricPrefix + '/' + metric.replace(".", "/"))
307361
.setAttributeFilter(attributesFilter)
308362
.build();
309363
viewMap.put(selector, view);
@@ -318,11 +372,7 @@ private static void defineGrpcGcpView(ImmutableMap.Builder<InstrumentSelector, V
318372
.setMeterName(BuiltInMetricsConstant.GRPC_GCP_METER_NAME)
319373
.build();
320374

321-
Set<String> attributesFilter =
322-
BuiltInMetricsConstant.COMMON_ATTRIBUTES.stream()
323-
.map(AttributeKey::getKey)
324-
.collect(Collectors.toSet());
325-
375+
Set<String> attributesFilter = baseAttributesFilter(false);
326376
attributesFilter.addAll(
327377
GRPC_GCP_METRIC_ADDITIONAL_ATTRIBUTES.getOrDefault(metric, ImmutableSet.of()));
328378

‎java-spanner/google-cloud-spanner/src/main/java/com/google/cloud/spanner/BuiltInMetricsProvider.java‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,16 @@ synchronized OpenTelemetry getOpenTelemetry() {
130130
return this.openTelemetry;
131131
}
132132

133+
/**
134+
* Injects the Cloud Monitoring OpenTelemetry so that {@link #getOrCreateOpenTelemetry} returns it
135+
* instead of building one backed by the live Cloud Monitoring exporter. Used to observe the
136+
* reserved-namespace sink in-memory in tests.
137+
*/
138+
@VisibleForTesting
139+
synchronized void setOpenTelemetry(OpenTelemetry openTelemetry) {
140+
this.openTelemetry = openTelemetry;
141+
}
142+
133143
synchronized String getProjectId() {
134144
return this.projectId;
135145
}
@@ -188,11 +198,22 @@ void enableGrpcMetrics(
188198
@Nullable Credentials credentials,
189199
@Nullable String monitoringHost,
190200
String universeDomain) {
201+
enableGrpcMetrics(
202+
channelProviderBuilder,
203+
this.getOrCreateOpenTelemetry(projectId, credentials, monitoringHost, universeDomain));
204+
}
205+
206+
/**
207+
* Wires gRPC-layer metrics into the channel builder using the given OpenTelemetry instance
208+
* instead of the process-wide Cloud Monitoring one. Used for {@link
209+
* CustomOpenTelemetryMetricsProvider}; does not touch any state of this singleton.
210+
*/
211+
void enableGrpcMetrics(
212+
InstantiatingGrpcChannelProvider.Builder channelProviderBuilder,
213+
OpenTelemetry openTelemetry) {
191214
GrpcOpenTelemetry grpcOpenTelemetry =
192215
GrpcOpenTelemetry.newBuilder()
193-
.sdk(
194-
this.getOrCreateOpenTelemetry(
195-
projectId, credentials, monitoringHost, universeDomain))
216+
.sdk(openTelemetry)
196217
.enableMetrics(BuiltInMetricsConstant.GRPC_METRICS_TO_ENABLE)
197218
// Disable gRPCs default metrics as they are not needed for Spanner.
198219
.disableMetrics(BuiltInMetricsConstant.GRPC_METRICS_ENABLED_BY_DEFAULT)

0 commit comments

Comments
 (0)