Skip to content

fix(tempo): enable local-blocks so traceql metrics queries work - #245

Merged
skyf0l merged 1 commit into
mainfrom
fix/tempo-traceql-metrics
Sep 13, 2026
Merged

fix(tempo): enable local-blocks so traceql metrics queries work#245
skyf0l merged 1 commit into
mainfrom
fix/tempo-traceql-metrics

Conversation

@skyf0l

@skyf0l skyf0l commented Sep 13, 2026

Copy link
Copy Markdown
Owner

What

Enables Tempo's metrics-generator with the local-blocks processor only, and
points its WAL and trace storage at the existing PVC.

Why

Grafana's Tempo explore page issues a TraceQL metrics query by default:

{nestedSetParent<0 && true && resource.service.name != nil} | rate() by(resource.service.name)

| rate() is computed by the metrics-generator, not by Prometheus. #242 left the
generator off, so Tempo logged "metrics-generator is not configured" at boot and
every such panel returned 500 with error finding generators: empty ring.

Only local-blocks is enabled — it answers TraceQL metrics from the generator's
own blocks. span-metrics and service-graphs stay off: they remote-write a
per-span-name and per-edge series set into Prometheus, and Thanos block size here
scales with distinct series, which already needed trimming once (#225).

filter_server_spans is set to false so rate() over an arbitrary TraceQL
selector matches what the trace view shows, rather than only server-kind spans.
Storage paths live under /var/tempo, which is the existing PVC, so generator
state survives a restart without a second volume.

Verification

  • Ran grafana/tempo:2.10.8 against the rendered config in Docker: modules
    metrics-generator-ring and metrics-generator both start and the instance
    registers in the ring. That is the direct cause of the reported error.
  • Note that -config.verify=true is NOT a meaningful gate on this version — it
    returned 0 for a deliberately invalid processor name and for an unknown config
    key, so the check above is a real run, not a lint.
  • make template: "all charts rendered OK". make validate-schema: 519 valid, 0
    invalid. prettier clean.

Risk

The generator receives a copy of every span in-process, so Tempo's memory use
rises; the pod limit stays at 1Gi and is worth watching once real traces flow.
flush_to_storage is left at its default of false, so this adds no R2 objects —
TraceQL metrics are answered from local blocks only, which bounds how far back
those queries can reach.

Rollback is reverting the commit; the generator's on-disk state is scratch.

Grafana's Tempo explore page issues a `| rate()` query by default, which is
served by the metrics-generator rather than Prometheus. With no generator in the
ring every such panel failed with "error finding generators: empty ring".

Only local-blocks is enabled. span-metrics and service-graphs remote-write a
per-span-name and per-edge series set into Prometheus, and Thanos block size here
scales with distinct series.
@skyf0l
skyf0l merged commit f9eeea7 into main Sep 13, 2026
4 checks passed
@skyf0l
skyf0l deleted the fix/tempo-traceql-metrics branch September 13, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant