Need help with ThanosRuler

[Migueljfs]

Hello everyone.

I'm deploying Thanos so we can manage prometheus scraping in two different clusters. One of these clusters is the "primary" one where I have:

- Thanos Query
- Thanos Query Frontend
- Thanos Store
- Thanos Compact
- Prometheus replicas with Thanos sidecar
- Alertmanager
- Grafana

On the "secondary" cluster I have only:

- Prometheus replicas with Thanos sidecar
- Alertmanager

In Thanos query:

- --store=dnssrv+_grpc._tcp.thanos-sidecar.monitoring.svc.cluster.local
- --store=dnssrv+_grpc._tcp.thanos-store.monitoring.svc.cluster.local
- --store=secondary-cluster.com:32490

All these is using GCS Buckets for storage. Both clusters write to the same bucket.

It all is working as expected, I have a "cluster" external label to separate both clusters. However I cannot understand how ThanosRuler is supposed to work here.

I created this manifest:

apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: evaluator
  namespace: monitoring
  labels:
    app.kubernetes.io/component: rule-evaluation-engine
    app.kubernetes.io/instance: thanos-rule
    app.kubernetes.io/name: thanos-rule
    app.kubernetes.io/version: v0.28.1
spec:
  image: quay.io/thanos/thanos:v0.28.1
  ruleSelector:
    matchLabels:
      role: alert-rules
  queryEndpoints:
    - http://thanos-query-frontend.monitoring:9090
  alertmanagersUrl:
    - http://alertmanager-main.monitoring.svc.cluster.local:9093

And applied it to the "primary" cluster but it does not grab alerts/rules from my "secondary" cluster.

What I expected?
I expected ThanosRuler to query my thanos-query-frontend, which will query my thanos-query, which will query my 3 stores configured, local sidecar, remote bucket and remote "secondary" cluster.

And then in Grafana I would be able to use ThanosRuler as a datasource to query alerts for all my clusters.

Do I have ThanosRuler architecture wrong?