Compactor halting repeatedly with fully overlapped blocks

[flaker]

I don't think there's really a bug here. But it's extremely consistent and I don't get the reason.

Thanos, Prometheus and Golang version used:

docker image: quay.io/thanos/thanos:v0.38.0

~ $ thanos --version
thanos, version 0.38.0 (branch: HEAD, revision: 0439da0dd291735c36bb515c03e9dfa0012c76c9)
  build user:       root@245568c15e20
  build date:       20250403-09:06:41
  go version:       go1.24.0
  platform:         linux/amd64
  tags:             netgo

Object Storage Provider:

S3

What happened:

I have 8 Prometheus instances configured via Prometheus Operator shipping blocks to S3 and one compactor taking care of all of them. Everything running in k8s.

I consistently run into problems with the exact same instance, same length of the blocks and the problematic blocks all claim to be created by the compactor. These are 2 of these blocks:

01K6YCJN79G05ZSB72XRWSEKXM

Start Time: October 3, 2024 5:00 PM
End Time: October 7, 2024 5:00 PM
Duration: 4 days
Series: 132123657
Samples: 41269556924
Chunks: 376024987
Total size: 129.58 GiB
Chunks: 111.45 GiB (86.01%)
Index: 18.12 GiB (13.99%)
Daily: 32.39 GiB / day
Resolution: 300000
Level: 4
Source: compactor
Labels:
prometheus: default/dai
prometheus_replica: prometheus-dai-0

01K4R6FM5P9M62EGEP1BY6A8NQ

Start Time: September 29, 2024 5:00 PM
End Time: October 9, 2024 5:00 PM
Duration: 10 days
Series: 291887188
Samples: 61837015304
Chunks: 666450110
Total size: 219.49 GiB
Chunks: 180.75 GiB (82.35%)
Index: 38.74 GiB (17.65%)
Daily: 21.95 GiB / day
Resolution: 300000
Level: 4
Source: compactor
Labels:
prometheus: default/dai
prometheus_replica: prometheus-dai-0

it's always the exact same thing. One block fully overlaps the other one, always 4 days and 10 days. Both blocks created by the compactor. I usually mark the smaller one for deletion, run deletion, restart the compactor.
 
What you expected to happen:

There should be no overlapping blocks.

How to reproduce it (as minimally and precisely as possible):

This only happens specifically with the blocks from this particular prometheus. The other 7 have never reported an issue.

Full logs to relevant components:

Logs

ts=2025-10-08T17:16:49.940446Z caller=factory.go:54 level=info msg="loading bucket configuration"
ts=2025-10-08T17:16:49.942715Z caller=verify.go:139 level=info verifiers=overlapped_blocks msg="Starting verify task"
ts=2025-10-08T17:16:49.943106Z caller=overlapped_blocks.go:30 level=info verifiers=overlapped_blocks verifier=overlapped_blocks msg="started verifying issue"
ts=2025-10-08T17:17:06.10799Z caller=fetcher.go:628 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=16.164814334s duration_ms=16164 cached=2131 returned=2103 partial=0
ts=2025-10-08T17:17:06.117769Z caller=overlapped_blocks.go:43 level=warn verifiers=overlapped_blocks verifier=overlapped_blocks msg="found overlapped blocks" group=300000@1955774452134216931 overlap="[mint: 1728000000000, maxt: 1728345600000, range: 96h0m0s, blocks: 2]: <ulid: 01K4R6FM5P9M62EGEP1BY6A8NQ, mint: 1727654400000, maxt: 1728518400000, range: 240h0m0s>, <ulid: 01K6YCJN79G05ZSB72XRWSEKXM, mint: 1728000000000, maxt: 1728345600000, range: 96h0m0s>"
ts=2025-10-08T17:17:06.117894Z caller=verify.go:158 level=info verifiers=overlapped_blocks msg="verify task completed"

(this log lines always look the same, just changing blocks)

Anything else we need to know:

This is the command line for Thanos in my template:

      compact
      --wait
      --log.level=debug
      --log.format=logfmt
      --objstore.config=$(OBJSTORE_CONFIG)
      --selector.relabel-config=$(RELABELING_CONFIG)
      --data-dir=/var/thanos/compact
      --retention.resolution-raw=0s
      --retention.resolution-5m=0s
      --retention.resolution-1h=0s
      --consistency-delay=30m
      --http-address=0.0.0.0:10902
      --compact.concurrency=2
      --downsample.concurrency=2

OBJSTORE_CONFIG points to an S3 bucket

RELABELING_CONFIG is set to:

- action: drop
  source_labels: ["prometheus"]
  regex: "default/dai-histograms"

since we do not want to process blocks coming from that particular prometheus (if there's a better way to do it, please share)

[GiedriusS]

Are writes succeeding? Perhaps you are running into #8401 ? Could you please try v0.40.0-rc.0 ?

[flaker]

as far as I know those writes are succedding (haven't seen anything in logs) and those are real working blocks. I'd like to somehow validate if those 2 blocks are complete and working but didn't find an option to do so in the version I am running. Something like

thanos tools bucket verify --issues=overlapped_blocks --objstore.config-file=objconfig.yaml --id [block id]

I will try to run a newer version.