fix(tempo): enable local-blocks so traceql metrics queries work - #245
Merged
Conversation
Grafana's Tempo explore page issues a `| rate()` query by default, which is served by the metrics-generator rather than Prometheus. With no generator in the ring every such panel failed with "error finding generators: empty ring". Only local-blocks is enabled. span-metrics and service-graphs remote-write a per-span-name and per-edge series set into Prometheus, and Thanos block size here scales with distinct series.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Enables Tempo's metrics-generator with the
local-blocksprocessor only, andpoints its WAL and trace storage at the existing PVC.
Why
Grafana's Tempo explore page issues a TraceQL metrics query by default:
| rate()is computed by the metrics-generator, not by Prometheus. #242 left thegenerator off, so Tempo logged "metrics-generator is not configured" at boot and
every such panel returned 500 with
error finding generators: empty ring.Only
local-blocksis enabled — it answers TraceQL metrics from the generator'sown blocks.
span-metricsandservice-graphsstay off: they remote-write aper-span-name and per-edge series set into Prometheus, and Thanos block size here
scales with distinct series, which already needed trimming once (#225).
filter_server_spansis set to false sorate()over an arbitrary TraceQLselector matches what the trace view shows, rather than only server-kind spans.
Storage paths live under
/var/tempo, which is the existing PVC, so generatorstate survives a restart without a second volume.
Verification
grafana/tempo:2.10.8against the rendered config in Docker: modulesmetrics-generator-ringandmetrics-generatorboth start and the instanceregisters in the ring. That is the direct cause of the reported error.
-config.verify=trueis NOT a meaningful gate on this version — itreturned 0 for a deliberately invalid processor name and for an unknown config
key, so the check above is a real run, not a lint.
make template: "all charts rendered OK".make validate-schema: 519 valid, 0invalid. prettier clean.
Risk
The generator receives a copy of every span in-process, so Tempo's memory use
rises; the pod limit stays at 1Gi and is worth watching once real traces flow.
flush_to_storageis left at its default of false, so this adds no R2 objects —TraceQL metrics are answered from local blocks only, which bounds how far back
those queries can reach.
Rollback is reverting the commit; the generator's on-disk state is scratch.