Smallest retention possible?

[nickaws]

I have this config in my compactor

I expect this to wipe from my s3 after a day, but it is not doing it. I might be misunderstanding, but I expect the data to disappear, but in grafana I still can go back and see everything. I don't see any errors in my compactor, and the data is still in S3. My data is a few kilobytes at most with a couple of metrics, it's a test instance.

        - compact
        - --wait
        - --log.level=debug
        - --log.format=logfmt
        - --objstore.config=$(OBJSTORE_CONFIG)
        - --data-dir=/var/thanos/compact
        - --debug.accept-malformed-index
        - --retention.resolution-raw=1d
        # - --retention.resolution-5m=10d
        # - --retention.resolution-1h=0d
        - --downsampling.disable 
        - --delete-delay=5m
        - --selector.relabel-config-file=/etc/config/relabel.yaml

I have this as my relabel config, the idea is to implement this retention mechanism

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: thanos2
  name: thanos-compact-relabel-1d
data:
  relabel.yaml: |
    - source_labels:
      - retention
      regex: "1d"
      action: keep
    - source_labels:
      - r
      regex: "1d"
      action: keep

thanos-compact-1d-0 thanos-compact ts=2024-11-09T03:50:53.394904634Z caller=blocks_cleaner.go:44 level=info msg="started cleaning of blocks marked for deletion"
thanos-compact-1d-0 thanos-compact ts=2024-11-09T03:50:53.394967517Z caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done"

[dosubot[bot]]

Hey there, @nickaws! I'm a bot designed to help you with bugs, questions, and contributions. Let's tackle this issue together while we wait for a human maintainer.

The issue you're experiencing, where data is not being deleted from S3 after one day despite your configuration, could be due to several factors:

1. Retention Configuration: Ensure that your retention settings are correctly configured. The --retention.resolution-raw=1d flag should theoretically delete raw resolution data after one day. However, if the other retention flags (--retention.resolution-5m and --retention.resolution-1h) are commented out, they default to no retention, meaning data at those resolutions will not be deleted.

2. Compaction and Downsampling: Since you have --downsampling.disable set, downsampling is disabled, which means only raw data is being considered for retention. Ensure that the compactor is running correctly and not halted due to errors such as overlapping blocks.

3. Compactor Halting: Check the thanos_compact_halted metric to see if the compactor is halted. This can happen due to overlapping blocks or other issues.

4. Concurrency and Resources: The compactor might be falling behind due to insufficient resources or concurrency settings. Consider increasing CPU and memory resources or adjusting the --compact.concurrency setting.

5. Backlog: Monitor the thanos_compact_iterations_total and thanos_compact_todo_deletion_blocks metrics to check for backlogs in processing and retention phases.

6. Grafana Caching: Sometimes, Grafana might cache data, making it appear as if the data is still available even after deletion. Ensure that Grafana is querying the correct time range and that there are no caching issues.

For more detailed troubleshooting, you can refer to the Thanos documentation on compactor backlog and troubleshooting ^[1][2].

To continue talking to Dosu, mention @dosu.

[yeya24]

Hi @nickaws,

this log means that your blocks should be marked for deletion now. Deletion is a lazy process so it takes some time for the blocks to be eventually deleted by the compactor. The delay by default is 48h. You can change delete-delay to a smaller value if you want.

thanos-compact-1d-0 thanos-compact ts=2024-11-09T03:50:53.394967517Z caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done"

[nickaws]

Thanks yeya! I have a delete day of 5 min. However I have blocks from several days.

[yeya24]

Can you check your blocks on S3 and see if they have deletion marker file uploaded? If so they should be deleted after

[nickaws]

They do, yes

deletion-mark.json | json | November 9, 2024, 20:20:18 (UTC-07:00) | 74.0 B | Standard

I'm using 0.36.1

[yeya24]

The deletion marker file should have a timestamp and the blocks are expected to be deleted after the timestamp. Can you please if it is already passing that time?

If the time is already passed then might be something wrong with your compactor

[nickaws]

I have re installed and no dice. Even if using the compactor UI, it does not delete blocks. Blocks are never marked for deletion nor there's any complain with the compactor.

In this case, I marked this block for deletion in the UI (block is 10h+ old), looks like no complain at all, but block still there (s3 permissions are s3:*)

thanos-compact-0 thanos-compact ts=2024-11-18T13:31:21.187174905Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=467.942362ms duration_ms=467 cached=5 returned=5 partial=0
thanos-compact-0 thanos-compact ts=2024-11-18T13:31:38.13408824Z caller=block.go:406 level=info msg="block has been marked for no compaction" block=01JCYV536SGSHF03RXVQ2XHEC3
thanos-compact-0 thanos-compact ts=2024-11-18T13:31:40.942245218Z caller=block.go:203 level=info msg="block has been marked for deletion" block=01JCYV536SGSHF03RXVQ2XHEC3
thanos-compact-0 thanos-compact ts=2024-11-18T13:32:20.719294126Z caller=fetcher.go:476 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
thanos-compact-0 thanos-compact ts=2024-11-18T13:32:21.184767366Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=465.482997ms duration_ms=465 cached=5 returned=5 partial=0
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.418780456Z caller=compact.go:1504 level=info msg="start sync of metas"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.419006206Z caller=fetcher.go:476 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.855192939Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=135.174685ms duration_ms=135 cached=5 returned=5 partial=0
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.858473511Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=439.465426ms duration_ms=439 cached=5 returned=0 partial=0
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.860691415Z caller=compact.go:1509 level=info msg="start of GC"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.862161821Z caller=compact.go:1532 level=info msg="start of compactions"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.862254319Z caller=compact.go:1568 level=info msg="compaction iterations done"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.86232765Z caller=compact.go:508 level=info msg="downsampling was explicitly disabled"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.864685593Z caller=fetcher.go:476 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.868590948Z caller=clean.go:34 level=info msg="started cleaning of aborted partial uploads"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.870407873Z caller=clean.go:61 level=info msg="cleaning of aborted partial uploads done"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.870440264Z caller=blocks_cleaner.go:44 level=info msg="started cleaning of blocks marked for deletion"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:20.870455788Z caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.009923498Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=145.239369ms duration_ms=145 cached=5 returned=0 partial=0
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.012111668Z caller=retention.go:32 level=info msg="start optional retention"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.01374288Z caller=retention.go:47 level=info msg="optional retention apply done"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.013956361Z caller=fetcher.go:476 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.152086679Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=138.103639ms duration_ms=138 cached=5 returned=0 partial=0
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.154530138Z caller=clean.go:34 level=info msg="started cleaning of aborted partial uploads"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.156119832Z caller=clean.go:61 level=info msg="cleaning of aborted partial uploads done"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.156193799Z caller=blocks_cleaner.go:44 level=info msg="started cleaning of blocks marked for deletion"
thanos-compact-0 thanos-compact ts=2024-11-18T13:33:21.156215169Z caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done"
thanos-compact-0 thanos-compact ts=2024-11-18T13:34:20.72016685Z caller=fetcher.go:476 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
thanos-compact-0 thanos-compact ts=2024-11-18T13:34:21.183748707Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=463.624995ms duration_ms=463 cached=5 returned=5 partial=0
thanos-compact-0 thanos-compact ts=2024-11-18T13:35:20.719211133Z caller=fetcher.go:476 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
thanos-compact-0 thanos-compact ts=2024-11-18T13:35:21.159657416Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=440.453593ms duration_ms=440 cached=5 returned=5 partial=0
thanos-compact-0 thanos-compact ts=2024-11-18T13:36:20.719867208Z caller=fetcher.go:476 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
thanos-compact-0 thanos-compact ts=2024-11-18T13:36:21.213066324Z caller=fetcher.go:623 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=493.200917ms duration_ms=493 cached=5 returned=5 partial=0

Receiver:

       - args:
        - receive
        - --log.level=debug
        - --log.format=logfmt
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --remote-write.address=0.0.0.0:19291
        - --receive.replication-factor=1
        - --tsdb.path=/var/thanos/receive
        - --tsdb.retention=3m
        - --label=replica="$(NAME)"
        - --label=receive="true"
        - --receive.local-endpoint=$(NAME).thanos-receive-ingestor.$(NAMESPACE).svc.cluster.local:10901
        # - --receive.hashrings-file=/var/lib/thanos-receive/hashrings.json
        - --receive.split-tenant-label-name="retention"
        # - --receive.tenant-label-name="retention"
        - --receive.default-tenant-id=3d
        - --label=ingestor="3d"
        - --objstore.config=$(OBJSTORE_CONFIG)

Compact

      - args:
        - compact
        - --wait
        - --log.level=debug
        - --log.format=logfmt
        - --objstore.config=$(OBJSTORE_CONFIG)
        - --data-dir=/var/thanos/compact
        - --debug.accept-malformed-index
        - --retention.resolution-raw=3h
        # - --retention.resolution-5m=0d
        # - --retention.resolution-1h=0d
        - --delete-delay=2m
        - --compact.concurrency=1
        - --downsample.concurrency=1
        - --downsampling.disable
        - --selector.relabel-config-file=/etc/config/relabel.yaml
        - --consistency-delay=1m
        # - --deduplication.replica-label=prometheus_replica
        # - --deduplication.replica-label=rule_replica
        # - |-
        #   --tracing.config="config":
        #     "sampler_param": 2
        #     "sampler_type": "ratelimiting"
        #     "service_name": "thanos-compact"
        #   "type": "JAEGER"

Relabel.yaml

  relabel.yaml: |
    - source_labels:
      - retention
      regex: "3d"
      action: keep