The issue of DNS resolution for the names of Thanos-related services returning "No answer"

[cscowx]

Hello, everyone!

Is it because these two domain names, my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local and my-thanos-store-headless.monitor.svc.cluster.local, cannot be properly dns resolved that the registration of the store and sidecar is not possible?
How should I troubleshoot this issue?

My pod/coredns-799cd8f49d-7xpvd is running normally.

I used nslookup for testing, and it often returned "No answer", but sometimes it would successfully resolve and return the IP address with very few attempts.

my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local" is a headless service.

kubectl exec -it pod/my-thanos-query-0 -n monitor -- nslookup -debug  my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Defaulted container "my-thanos-query" out of: my-thanos-query, config-render (init)
Server:         10.96.0.2
Address:        10.96.0.2:53

Query #0 completed in 2ms:
authoritative answer:

*** Can't find my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local: No answer


#The log output of this coredns entry

kubectl logs -f pod/coredns-799cd8f49d-7xpvd -n kube-system|grep my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local


[INFO] 10.244.218.135:35599 - 39168 "AAAA IN my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local. udp 81 false 512" NOERROR qr,aa,rd 174 0.000329277s
[INFO] 10.244.218.135:35599 - 39168 "A IN my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local. udp 81 false 512" NOERROR qr,aa,rd 239 0.000427503s
[INFO] 10.244.218.135:35599 - 39168 "AAAA IN my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local. udp 81 false 512" NOERROR qr,aa,rd 174 0.000302363s

kubectl exec -it pod/my-thanos-query-0 -n monitor -- nslookup -debug  my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Defaulted container "my-thanos-query" out of: my-thanos-query, config-render (init)
Server:         10.96.0.2
Address:        10.96.0.2:53

Query #0 completed in 1ms:
authoritative answer:
Name:   my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Address: 10.244.156.200
Name:   my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Address: 10.244.218.134

*** Can't find my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local: No answer

#The log output of this coredns entry

kubectl logs -f pod/coredns-799cd8f49d-7xpvd -n kube-system|grep my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local

[INFO] 10.244.218.135:49827 - 37632 "A IN my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local. udp 81 false 512" NOERROR qr,aa,rd 239 0.000251902s
[INFO] 10.244.218.135:49827 - 37632 "AAAA IN my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local. udp 81 false 512" NOERROR qr,aa,rd 174 0.000416981s
[INFO] 10.244.218.135:49827 - 37632 "AAAA IN my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local. udp 81 false 512" NOERROR qr,aa,rd 174 0.00046752s

I tested it using wget and it should be able to parse out the IP address.

kubectl exec -it pod/my-thanos-query-0 -n monitor -- wget my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Defaulted container "my-thanos-query" out of: my-thanos-query, config-render (init)
Connecting to my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local (10.244.156.200:80)
wget: can't connect to remote host (10.244.156.200): Connection refused
command terminated with exit code 1

kubectl exec -it pod/my-thanos-query-0 -n monitor -- wget my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Defaulted container "my-thanos-query" out of: my-thanos-query, config-render (init)
Connecting to my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local (10.244.218.134:80)
wget: can't connect to remote host (10.244.218.134): Connection refused
command terminated with exit code 1

I conducted a test using other pods within the same monitor namespace, and found that DNS resolution could respond normally very quickly. As follows:

kubectl exec -it pod/test-source-100-587d68bc6c-2gtk2 -n monitor -- nslookup -debug my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
;; Got recursion not available from 10.96.0.2
Server:         10.96.0.2
Address:        10.96.0.2#53

------------
    QUESTIONS:
        my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local.monitor.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
        origin = ns.dns.cluster.local
        mail addr = hostmaster.cluster.local
        serial = 1760956989
        refresh = 7200
        retry = 1800
        expire = 86400
        minimum = 60
        ttl = 60
    ADDITIONAL RECORDS:
------------
** server can't find my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local.monitor.svc.cluster.local: NXDOMAIN
;; Got recursion not available from 10.96.0.2
Server:         10.96.0.2
Address:        10.96.0.2#53

------------
    QUESTIONS:
        my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
        origin = ns.dns.cluster.local
        mail addr = hostmaster.cluster.local
        serial = 1760956989
        refresh = 7200
        retry = 1800
        expire = 86400
        minimum = 60
        ttl = 60
    ADDITIONAL RECORDS:
------------
** server can't find my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local.svc.cluster.local: NXDOMAIN
;; Got recursion not available from 10.96.0.2
Server:         10.96.0.2
Address:        10.96.0.2#53

------------
    QUESTIONS:
        my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
        origin = ns.dns.cluster.local
        mail addr = hostmaster.cluster.local
        serial = 1760956989
        refresh = 7200
        retry = 1800
        expire = 86400
        minimum = 60
        ttl = 60
    ADDITIONAL RECORDS:
------------
** server can't find my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local.cluster.local: NXDOMAIN
;; Got recursion not available from 10.96.0.2
Server:         10.96.0.2
Address:        10.96.0.2#53

------------
    QUESTIONS:
        my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local, type = A, class = IN
    ANSWERS:
    ->  my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
        internet address = 10.244.218.134
        ttl = 60
    ->  my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
        internet address = 10.244.156.200
        ttl = 60
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Address: 10.244.218.134
Name:   my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Address: 10.244.156.200
;; Got recursion not available from 10.96.0.2
------------
    QUESTIONS:
        my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local, type = AAAA, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
        origin = ns.dns.cluster.local
        mail addr = hostmaster.cluster.local
        serial = 1760956989
        refresh = 7200
        retry = 1800
        expire = 86400
        minimum = 60
        ttl = 60
    ADDITIONAL RECORDS:
------------

command terminated with exit code 1

My pod/my-thanos-query-0 is connected to these two Pods through the two headless services, my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local and my-thanos-store-headless.monitor.svc.cluster.local. Please refer to the following configuration of the thanos query for details.

#Detailed configuration details
- --endpoint.sd-config-file=/etc/thanos/endpoints.json

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Service
metadata:
  name: my-thanos-query
  namespace: monitor
spec:
  ports:
    - port: 10901
      targetPort: 10901
      name: grpc
    - port: 10902
      targetPort: 10902
      name: http
      nodePort: 30902
  selector:
    app: my-thanos-query
  publishNotReadyAddresses: true
  type: NodePort
EOF


cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-thanos-query-request-logging-config
  namespace: monitor
data:
  request-logging.yaml: |
    http:
      options:
        level: INFO
        decision:
          log_start: true
          log_end: true
    grpc:
      options:
        level: INFO
        decision:
          log_start: true
          log_end: true
EOF

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-thanos-endpoints-config
  namespace: monitor
data:
  endpoints.json: |
    {
      "endpoints": [
        {
          "labels": {
            "group": "my-prometheus-thanos-sidecar",
            "env": "prod",
            "cluster": "nanjing-k8s"
          },
          "dns_sd_configs": [
            {
              "names": ["my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local"],
              "type": "A",
              "port": 10901
            }
          ]
        },
        {
          "labels": {
            "group": "my-thanos-store",
            "env": "prod",
            "cluster": "nanjing-k8s"
          },
          "dns_sd_configs": [
            {
              "names": ["my-thanos-store-headless.monitor.svc.cluster.local"],
              "type": "A",
              "port": 10901
            }
          ]
        }
      ]
    }
EOF

cat <<'EOF' | kubectl apply -f -
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-thanos-query
  namespace: monitor
spec:
  selector:
    matchLabels:
      app: my-thanos-query
  serviceName: "my-thanos-query"
  replicas: 2
  template:
    metadata:
      labels:
        app: my-thanos-query
    spec:
      terminationGracePeriodSeconds: 60
      dnsPolicy: ClusterFirst
      serviceAccountName: my-prometheus
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: my-thanos-query
                topologyKey: kubernetes.io/hostname
      initContainers:
        - name: config-render
          image: m.daocloud.io/docker.io/bash:latest
          securityContext:
            runAsUser: 0
            runAsNonRoot: false
            allowPrivilegeEscalation: false
          env:
            - name: REPLICA_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          command:
            - /bin/sh
            - -c
            - |
              set -eux && \
              printenv >&2 && echo "REPLICA_NAME=${REPLICA_NAME}" >&2 && \
              mkdir -p /mnt/${REPLICA_NAME}/data && \
              mkdir -p /mnt/${REPLICA_NAME}/logs && \
              cp -rp /tmp/my-thanos-storage-config.yml /mnt/${REPLICA_NAME}/ && \
              chown -R 1000: /mnt
          volumeMounts:
              - name: my-thanos-storage-configmap
                subPath: my-thanos-storage-config.yml
                mountPath: /tmp/my-thanos-storage-config.yml
                readOnly: true
              - name: nfs-volume-prometheus-cfg
                mountPath: /mnt
                readOnly: false
      containers:
        - name: my-thanos-query
          image: m.daocloud.io/docker.io/thanosio/thanos:main-2025-10-07-be442a5
          imagePullPolicy: "IfNotPresent"
          env:
            - name: REPLICA_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          args:
            - query 
            - --log.level=warn 
            - --log.format=json 
            - --enable-auto-gomemlimit 
            - --auto-gomemlimit.ratio=0.8 
            - --http-address=0.0.0.0:10902 
            - --http-grace-period=30s 
            - --grpc-address=0.0.0.0:10901
            - --grpc-grace-period=30s 
            - --grpc-server-max-connection-age=30m 
            - --web.external-prefix=/thanos/query 
            - --web.route-prefix=/thanos/query 
            - --query.timeout=2m 
            - --query.max-concurrent=20 
            - --query.max-concurrent-select=8 
            - --query.auto-downsampling 
            - --query.partial-response 
            - --query.lookback-delta=5m 
            - --query.replica-label=replica 
            - --enable-feature=promql-experimental-functions 
            - --query.default-step=15s 
            - --store.response-timeout=30s 
            - --store.limits.request-series=5000 
            - --store.limits.request-samples=26000000 
            - --store.unhealthy-timeout=5m 
            - --store.sd-dns-interval=30s
            - --endpoint.sd-config-reload-interval=5m 
            - --query.active-query-path=/etc/thanos/logs/queries.active 
            - --request.logging-config-file=/etc/thanos/request-logging.yaml
            - --alert.query-url=http://my-thanos-query.monitor.svc.cluster.local:10902/thanos/query
            - --endpoint.sd-config-file=/etc/thanos/endpoints.json
          ports:
            - containerPort: 10902
              name: query-http
            - containerPort: 10901
              name: query-grpc
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
          startupProbe:
            exec:
              command:
                - hostname
            initialDelaySeconds: 60
            periodSeconds: 15
            successThreshold: 1
            failureThreshold: 6
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: query-http
            initialDelaySeconds: 120
            periodSeconds: 15
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /-/ready
              port: query-http
              scheme: HTTP
            initialDelaySeconds: 60
            periodSeconds: 15
            successThreshold: 1
            failureThreshold: 3
          resources:
            requests:
              cpu: 1
              memory: 1024Mi
            limits:
              cpu: 1
              memory: 1024Mi
          volumeMounts:
              - name: nfs-volume-prometheus-cfg
                mountPath: /etc/thanos/logs/
                subPathExpr: $(REPLICA_NAME)/logs
                readOnly: false
              - name: my-thanos-query-request-logging-config
                subPath: request-logging.yaml
                mountPath: /etc/thanos/request-logging.yaml
                readOnly: true
              - name: my-thanos-endpoints-config
                subPath: endpoints.json
                mountPath: /etc/thanos/endpoints.json
                readOnly: true
              - name: ssl
                mountPath: /ssl
                readOnly: true
      volumes:
        - name: ssl
          hostPath:
            path: /ssl
        - name: nfs-volume-prometheus-cfg
          nfs:
            server: 192.168.110.220
            path: /nfs-data/prometheus-cfg
        - name: my-thanos-storage-configmap
          configMap:
            name: my-thanos-storage-config
            items:
              - key: my-thanos-storage-config.yml
                path: my-thanos-storage-config.yml
        - name: my-thanos-query-request-logging-config
          configMap:
            name: my-thanos-query-request-logging-config
            items:
              - key: request-logging.yaml
                path: request-logging.yaml
        - name: my-thanos-endpoints-config
          configMap:
            name: my-thanos-endpoints-config
            items:
              - key: endpoints.json
                path: endpoints.json
EOF

[cscowx]

My CoreDNS does not support IPv6.

I found that for type A, normal DNS resolution is possible.

kubectl exec -it pod/my-thanos-query-0 -n monitor -- nslookup -type=A my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Defaulted container "my-thanos-query" out of: my-thanos-query, config-render (init)
Server:         10.96.0.2
Address:        10.96.0.2:53

Name:   my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Address: 10.244.218.134
Name:   my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Address: 10.244.156.200

I found that in the following form, normal DNS resolution is not possible.

kubectl exec -it pod/my-thanos-query-0 -n monitor -- nslookup my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Defaulted container "my-thanos-query" out of: my-thanos-query, config-render (init)
Server:         10.96.0.2
Address:        10.96.0.2:53
*** Can't find my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local: No answer

kubectl exec -it pod/my-thanos-query-0 -n monitor -- nslookup -type=AAAA my-prometheus-thanos-sidecar-headless.monitor.svc.cluster.local
Defaulted container "my-thanos-query" out of: my-thanos-query, config-render (init)
Server:         10.96.0.2
Address:        10.96.0.2:53

kubectl get statefulsets,deployments,pods,service -n monitor

NAME                                            READY   AGE
statefulset.apps/my-prometheus-thanos-sidecar   2/2     4d
statefulset.apps/my-thanos-query                2/2     4d16h
statefulset.apps/my-thanos-store                2/2     4d21h

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-blackbox-exporter   1/1     1            1           21d
deployment.apps/test-source-100        1/1     1            1           3d16h
deployment.apps/test-source-200        1/1     1            1           3d16h

NAME                                        READY   STATUS    RESTARTS      AGE
pod/my-blackbox-exporter-6ccd9b9f57-qwbdj   1/1     Running   1 (23h ago)   4d16h
pod/my-prometheus-thanos-sidecar-0          2/2     Running   0             17h
pod/my-prometheus-thanos-sidecar-1          2/2     Running   0             17h
pod/my-thanos-query-0                       1/1     Running   0             13h
pod/my-thanos-query-1                       1/1     Running   0             13h
pod/my-thanos-store-0                       1/1     Running   5 (23h ago)   4d21h
pod/my-thanos-store-1                       1/1     Running   1 (23h ago)   4d16h
pod/test-source-100-587d68bc6c-2gtk2        1/1     Running   1 (23h ago)   3d16h
pod/test-source-200-68c79f4bcf-9qrqf        1/1     Running   1 (23h ago)   3d16h

NAME                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                           AGE
service/my-blackbox-exporter                    ClusterIP   10.110.191.46    <none>        9115/TCP                          26d
service/my-prometheus-service                   NodePort    10.105.199.75    <none>        9090:39090/TCP                    29d
service/my-prometheus-thanos-sidecar-headless   ClusterIP   None             <none>        10901/TCP,10902/TCP               4d17h
service/my-thanos-query                         NodePort    10.107.201.235   <none>        10901:35678/TCP,10902:30902/TCP   4d16h
service/my-thanos-store-headless                ClusterIP   None             <none>        10901/TCP,10902/TCP               4d21h
service/test-source-100                         ClusterIP   None             <none>        10901/TCP,10902/TCP               3d16h
service/test-source-200                         ClusterIP   None             <none>        10901/TCP,10902/TCP               3d16h

Querying the Pod of the thanos query component, the following error logs were found:

kubectl logs -f pod/my-thanos-query-0 -n monitor

{"addr":"","caller":"endpointset.go:240","level":"warn","msg":"Duplicate endpoint address is provided","ts":"2025-10-21T02:24:28.041185203Z"}
{"address":"","caller":"endpointset.go:407","component":"endpointset","err":"getting metadata: failed to exit idle mode: dns resolver: missing address","level":"warn","msg":"update of endpoint failed","ts":"2025-10-21T02:24:28.041727177Z"}
{"addr":"","caller":"endpointset.go:240","level":"warn","msg":"Duplicate endpoint address is provided","ts":"2025-10-21T02:24:33.03770902Z"}
{"address":"","caller":"endpointset.go:407","component":"endpointset","err":"getting metadata: failed to exit idle mode: dns resolver: missing address","level":"warn","msg":"update of endpoint failed","ts":"2025-10-21T02:24:33.038400911Z"}

What should I do to resolve this error log here?

Additionally, I'm not sure if the configuration in the above section regarding --endpoint.sd-config-file=/etc/thanos/endpoints.json is in the correct format. I haven't found any specific configuration method on the official website either. Please correct me if I'm wrong.