-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Description
Client
Firestore sets trace span names with id in them, resulting in polluting traces with garbage and high cardinality and cost. you would get something like /my_collection/_doc/asdf1234 traces of kind client
this is very bad, I just found out our spants are polluted with this garbage produced by official GCP Go Firestore client. this is so bad, I have to shutdown whole firestore telemetry, because it is a garbage firehose into our telemetry.
trace spans MUST NOT
contain high cardinality IDs. the purpose of span is to group repeated invocations to track their: latency; error rate; volume of requests. now, if we have unique IDs there.. this does not make sense, likelihood of hitting same ID is very low and it is not useful.... but whom I am explaning this all about? you guys written "Site Reliability at Google" and pionireed SRE, right?
please fix this!
Environment
- Alpine Docker on GKE
- go 1.25
Code and Dependencies
we are using OpenTelemetry and GCP Cloud Monitoring as a sink for traces. Grafana dashbaord based on span_metrics OTEL Plugin: https://github.com/ndx-technologies/grafana-open-telemetry-apm + https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector
go.mod
cloud.google.com/go/firestore v1.18.0
...
go.opentelemetry.io/contrib/detectors/gcp v1.38.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0
go.opentelemetry.io/contrib/instrumentation/runtime v0.63.0
go.opentelemetry.io/otel v1.38.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.38.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.38.0
go.opentelemetry.io/otel/metric v1.38.0
go.opentelemetry.io/otel/sdk v1.38.0
go.opentelemetry.io/otel/sdk/metric v1.38.0
go.opentelemetry.io/otel/trace v1.38.0
Expected behavior
no high cardniality IDs in trace span names. only in labels.
Actual behavior
high cardinality IDs in trace span names.
Screenshots

Additional context
N/A