Add configurable limit for postings expansion in store-gateway /labels requests

[dimitarvdimitrov]

Problem

Store-gateway /labels requests with broad matchers (e.g., {__name__!=""}) can cause OOM kills by consuming excessive memory during postings expansion. This was observed in production where:

• A single /labels request consumed ~10Gi memory expanding ~1 billion postings (8 bytes each)
• Multiple concurrent requests can exhaust store-gateway memory and crash the process

The memory allocation occurs in bucketIndexReader.expandedPostings() when calling index.ExpandPostings(result), which loads all matching postings into memory before filtering label values.

Proposed Solution

Mainly proposed by @bboreham

Add a configurable limit on memory usage per request in store-gateway, similar to the ingester's -ingester.read-path-memory-utilization-limit. Initially this will track postings memory usage, but can be extended to track other memory allocations as reactive limiters are implemented.

Configuration:

• New flag: -blocks-storage.bucket-store.max-memory-bytes-per-request: percentage of GOMEMLIMIT
• Suggested default: 10% of GOMEMLIMIT (e.g., 600M postings for 48Gi store-gateway = 4.8Gi limit)

Behavior:

• Return gRPC status code ResourceExhausted when limit would be exceeded, rather than OOMing
• Error message should indicate the query is too broad and suggest more specific matchers

Implementation approach:

• Implement wrapper iterator around postings expansion (analogous to io.LimitedReader)
• Add limit check in bucketIndexReader.expandedPostings() before calling index.ExpandPostings()
• Initially could be logging-only to measure impact before enforcing

Consider starting with a warning-only mode to gather metrics on how often the limit would be hit before enforcing it.

[56quarters]

Note that MQE added a MemoryConsumptionTracker for enforcing per-request memory limits that may work for this purpose. I don't know how/if it transits gRPC boundaries but it might be possible to link this memory usage back to the per-request limit that MQE uses It doesn't get sent with the gRPC request but it should still be possible to use it for per-request limits within the store-gateway.

[dimitarvdimitrov]

if we can propagate this it would be great

But we'd still need some way to tie the per-query consumption with a per-pod limit. I mean some way to say that whatever the memory limit of the query is, it still can't use more than 10% of a store-gateway (or all in-queries summed together in the store-gateway can't use more than 100% of the GOMEMLIMIT)

[charleskorn]

But we'd still need some way to tie the per-query consumption with a per-pod limit. I mean some way to say that whatever the memory limit of the query is, it still can't use more than 10% of a store-gateway (or all in-queries summed together in the store-gateway can't use more than 100% of the GOMEMLIMIT)

Can we use some Jsonnet magic to set a per-query limit with a CLI flag on the store-gateways with the limit based on the memory request for the pod? Then we have the flexibility to override that value with whatever we want if that value turns out to be problematic (either too high or too low).

Or should we make this configurable per tenant as well?

[dimitarvdimitrov]

Do you prefer jsonnet magic to have a flag with a %? -blocks-storage.bucket-store.max-memory-per-query=[0.0, 1.0] which would set this up based on GOMEMLIMIT?

I see this mostly as a limit to protect the store-gateway pod. And less as a way of having fair usage between tenants of different sizes.

[charleskorn]

Do you prefer jsonnet magic to have a flag with a %? -blocks-storage.bucket-store.max-memory-per-query=[0.0, 1.0] which would set this up based on GOMEMLIMIT?

Thinking about this more, I think I prefer an absolute value, rather than a percentage of the store-gateway's memory request. The reason for this is if we use a percentage, then a request that succeeds today may fail tomorrow if the store-gateway's memory request is decreased, which doesn't seem like a great user experience.

[dimitarvdimitrov]

fail tomorrow if the store-gateway's memory request is decreased

realistically with automated vertical scaling and jsonnet magic it would be the same