Thanos Ruler: got 401 Body: Unauthorized

[JoshuaSmeda]

Context:

I'm busy setting up a Thanos stack. So far, things have been smooth sailing.

I've deployed the stack using a Helm Chart

alertmanager-kube-prometheus-stack-0                        Running
kube-prometheus-stack-grafana-6b79f9bd7f-kdmc4              Running
kube-prometheus-stack-kube-state-metrics-656b554d59-xdvc2   Running
kube-prometheus-stack-operator-67b889c86d-pqpv7             Running
kube-prometheus-stack-prometheus-node-exporter-bz6pg        Running
kube-prometheus-stack-prometheus-node-exporter-cswqj        Running
kube-prometheus-stack-prometheus-node-exporter-f2l5h        Running
kube-prometheus-stack-prometheus-node-exporter-vpcff        Running
prometheus-kube-prometheus-stack-0                          Running
thanos-compactor-7fbf798589-n8tfp                           Running
thanos-query-75fd889f4-tgnfh                                Running
thanos-query-frontend-6f476cdf4b-2t2gn                      Running
thanos-receive-0                                            Running
thanos-receive-1                                            Running
thanos-ruler-0                                              Running
thanos-storegateway-0                                       Running

I've added basic authentication:

auth:
  basicAuthUsers:
    xxx: xxx

which generates something similar to this:

k get secret thanos-http-config-secret -o json | jq '.data | map_values(@base64d)'
{
  "http-config.yml": "basic_auth_users:\n  xxx: xxx"
}

which is injected into all my pods.

My problem:

I cannot get my Thanos Ruler to authenticate to my Thanos query API.

The error I receive:

ts=2024-05-10T15:35:17.100967998Z caller=promclient.go:437 level=debug msg="querying instant" url="http://172.20.112.174:9090/api/v1/query?analyze=false&dedup=true&engine=&explain=false&partial_response=false&query=absent%28up%7Bprometheus%3D%22monitoring%2Fkube-prometheus-stack%22%7D%29&time=2024-05-10T15%3A35%3A17.0967074Z"
ts=2024-05-10T15:35:17.10288841Z caller=rule.go:926 level=error component=rules err="read query instant response: expected 2xx response, got 401. Body: Unauthorized\n" query="absent(up{prometheus=\"monitoring/kube-prometheus-stack\"})"

My arguments passed into the Ruler container, this worked perfectly fine without authentication:

    Args:
      rule
      --log.level=debug
      --log.format=logfmt
      --grpc-address=0.0.0.0:10901
      --http-address=0.0.0.0:10902
      --data-dir=/data
      --eval-interval=1m
      --http.config=/conf/http/http-config.yml
      --alertmanagers.url=http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093
      --query=dnssrv+_http._tcp.thanos-query.monitoring.svc.cluster.local
      --alert.query-url=http://thanos-query.monitoring.svc.cluster.local:9090
      --label=replica="$(POD_NAME)"
      --label=ruler_cluster=""
      --alert.label-drop=replica
      --objstore.config-file=/conf/objstore/objstore.yml
      --rule-file=/conf/rules/*.yml

The query and http.config are passed in via the Helm chart. So I can't manually specify an additional flag of http.config because of duplicate keys, which could include my basic_auth credentials as described here.

How am I supposed to authenticate to the query API? Is there a simpler way that I may be missing?

And why is there a disparity between the different Thanos components? Why does the query API leverage basic_auth and the rest of the components require basic_auth_users configuration?

I hope I explained my problem clearly enough. If you have any comments, questions, or suggestions, please do reply.

[mkozikowsk]

Hi @JoshuaSmeda, I had similar issue with ThanosRuler authentication.
You have to provide custom query config for Thanos Ruler by using additional flag --query.config-file
https://thanos.io/tip/components/rule.md/#query-api

If you want to configure that in helm chart you have to do a workaround by editing ruler values like that:

  ruler:
    extraVolumeMounts:
    - name: query-config
      mountPath: /mnt/query-config
      
    extraVolumes:
    - name: query-config
      secret:
        defaultMode: 420
        secretName: thanos-ruler-query-config
        
    extraFlags:
      - '--query.config-file=/mnt/query-config.yaml'
      
    dnsDiscovery:
      enabled: false

and secret with query-config should look like this:

apiVersion: v1
kind: Secret
metadata:
  name: thanos-ruler-query-config
  namespace: thanos
stringData:
  query-config.yaml: |-
    - static_configs:
      - "thanos-query.thanos:9090"
      scheme: http
      http_config:
        basic_auth:
          username: admin
          password: password