Skip to content

firestore: trace span names contain high cardinality ids #12973

@nikolaydubina

Description

@nikolaydubina

Client

Firestore sets trace span names with id in them, resulting in polluting traces with garbage and high cardinality and cost. you would get something like /my_collection/_doc/asdf1234 traces of kind client

this is very bad, I just found out our spants are polluted with this garbage produced by official GCP Go Firestore client. this is so bad, I have to shutdown whole firestore telemetry, because it is a garbage firehose into our telemetry.

trace spans MUST NOT contain high cardinality IDs. the purpose of span is to group repeated invocations to track their: latency; error rate; volume of requests. now, if we have unique IDs there.. this does not make sense, likelihood of hitting same ID is very low and it is not useful.... but whom I am explaning this all about? you guys written "Site Reliability at Google" and pionireed SRE, right?

please fix this!

Environment

  • Alpine Docker on GKE
  • go 1.25

Code and Dependencies

we are using OpenTelemetry and GCP Cloud Monitoring as a sink for traces. Grafana dashbaord based on span_metrics OTEL Plugin: https://github.com/ndx-technologies/grafana-open-telemetry-apm + https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector

go.mod

cloud.google.com/go/firestore v1.18.0
...
	go.opentelemetry.io/contrib/detectors/gcp v1.38.0
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0
	go.opentelemetry.io/contrib/instrumentation/runtime v0.63.0
	go.opentelemetry.io/otel v1.38.0
	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.38.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.38.0
	go.opentelemetry.io/otel/metric v1.38.0
	go.opentelemetry.io/otel/sdk v1.38.0
	go.opentelemetry.io/otel/sdk/metric v1.38.0
	go.opentelemetry.io/otel/trace v1.38.0

Expected behavior

no high cardniality IDs in trace span names. only in labels.

Actual behavior

high cardinality IDs in trace span names.

Screenshots

Image

Additional context

N/A

Metadata

Metadata

Assignees

Labels

triage meI really want to be triaged.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions