What is the recommended way of configuring kube-proxy metrics scraping?

[gerhard]

What is the problem?

kube-prometheus-stack requires a non-loopback IP for scraping metrics.

Scraping http://localhost:10249/metrics works, but http://<NODE_IP>:10249/metrics doesn't.

kube-proxy listens on 127.0.0.1:

# netstat -ltnp | grep kube-proxy
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      212123/kube-proxy
tcp6       0      0 :::10256                :::*                    LISTEN      212123/kube-proxy

The pod advertises <NODE_IP>:

# kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide
NAME               READY   STATUS    RESTARTS   AGE   IP              NODE
kube-proxy-9cdcq   1/1     Running   0          22h   192.168.88.12   h12-london

I am wondering if it's recommended to patch ClusterConfig.proxy with:

extraArgs:
  bind-address: '0.0.0.0'

What is the current setup?

I am running Talos v1.5.3 managed by Omni v0.18.1 with K8s v1.28.2 & kube-prometheus-stack v48.3.6

This is a single-node cluster, with scheduling enabled on the control plane node.

This is the kube-prometheus-stack values excerpt that should enable kube-proxy scraping:

# ↑ 1738 rows of config before
kubeProxy:
  enabled: true

  service:
    enabled: true
    port: 10249
    targetPort: 10249
    selector:
      k8s-app: kube-proxy
# ↓ 2300 rows of config after

The kube-prometheus-stack-kube-proxy headless service gets created accurately, based on the kube-proxy pod:

# kubectl describe service/kube-prometheus-stack-kube-proxy -n kube-system
Name:              kube-prometheus-stack-kube-proxy
Namespace:         kube-system
Labels:            app=kube-prometheus-stack-kube-proxy
                   app.kubernetes.io/instance=kube-prometheus-stack
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/part-of=kube-prometheus-stack
                   app.kubernetes.io/version=48.3.6
                   chart=kube-prometheus-stack-48.3.6
                   heritage=Helm
                   jobLabel=kube-proxy
                   release=kube-prometheus-stack
Annotations:       meta.helm.sh/release-name: kube-prometheus-stack
                   meta.helm.sh/release-namespace: kube-prometheus-stack
Selector:          k8s-app=kube-proxy
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              http-metrics  10249/TCP
TargetPort:        10249/TCP
Endpoints:         192.168.88.12:10249
Session Affinity:  None
Events:            <none>

And this is what the kube-proxy pod looks like:

# kubectl describe pods -n kube-system -l k8s-app=kube-proxy
Name:                 kube-proxy-9cdcq
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      kube-proxy
Node:                 h12-london/192.168.88.12
Start Time:           Wed, 27 Sep 2023 12:01:00 +0100
Labels:               controller-revision-hash=669cd96d9
                      k8s-app=kube-proxy
                      pod-template-generation=2
                      tier=node
Annotations:          <none>
Status:               Running
IP:                   192.168.88.12
IPs:
  IP:           192.168.88.12
Controlled By:  DaemonSet/kube-proxy
Containers:
  kube-proxy:
    Container ID:  containerd://414590226d0267b6e63d54d55e90eba88ccf867312cfc2e2f6159cfc2cc27fcb
    Image:         registry.k8s.io/kube-proxy:v1.28.2
    Image ID:      registry.k8s.io/kube-proxy@sha256:41c8f92d1cd571e0e36af431f35c78379f84f5daf5b85d43014a9940d697afcf
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/kube-proxy
      --cluster-cidr=10.244.0.0/16
      --conntrack-max-per-core=0
      --hostname-override=$(NODE_NAME)
      --kubeconfig=/etc/kubernetes/kubeconfig
      --proxy-mode=iptables
    State:          Running
      Started:      Wed, 27 Sep 2023 12:01:05 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
      POD_IP:      (v1:status.podIP)
    Mounts:
      /etc/kubernetes from kubeconfig (ro)
      /etc/ssl/certs from ssl-certs-host (ro)
      /lib/modules from lib-modules (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bjw9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  ssl-certs-host:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:
  kubeconfig:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubeconfig-in-cluster
    Optional:  false
  kube-api-access-9bjw9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:                      <none>

What is the question again?

What is the recommended way of configuring kube-proxy metrics scraping?

[gerhard]

I have a similar question, but for etcd, which is running outside of K8s:

• How to get etcd metrics? #7214 (comment)

[smira]

@TimJones might correct me if I'm wrong, but I believe the only way with kube-proxy is to make it bind to *.

[gerhard]

That's what I'm thinking too, just checking that this is indeed the recommended way.

Not sure how secure it is to make kube-proxy available on all interfaces.

[TimJones]

kube-proxy has a metrics-bind-address arg so you can specify IP & port on that, like metrics-bind-address: 0.0.0.0:10249.

[gerhard]

Thanks, will try this out!

[gerhard]

Works! Thank you @TimJones & @smira 🙌

kube-prometheus-stack.yaml

# ↑ 1737 rows of config before

## Component scraping kube proxy
##
kubeProxy:
  enabled: true

  ## If your kube proxy is not deployed as a pod, specify IPs it can be found on
  ##
  endpoints: []

  service:
    enabled: true
    port: 10249
    targetPort: 10249
    selector:
      k8s-app: kube-proxy

# ↓ 2301 lines of config after

patches/kube-proxy-metrics.yaml

# https://github.com/siderolabs/talos/discussions/7799#discussioncomment-7135226
# ⚠️ Requires `omnictl cluster kubernetes manifest-sync $CLUSTER --dry-run=false` after changing!
cluster:
  proxy:
    extraArgs:
      metrics-bind-address: 0.0.0.0:10249

[vitali-ipquants]

kube-proxy has a metrics-bind-address arg so you can specify IP & port on that, like metrics-bind-address: 0.0.0.0:10249.

Hello @TimJones ,
I recreated the control plane nodes with patched manifest containing /cluster/proxy/extraArgs/metrics-bind-address: 0.0.0.0:10249.
I can see the change in the output of talosctl -n <cp-node-ip> get machineconfig -o yaml,
but the definition of the kube-proxy daemonset is not updated.
What's the way to sync the cluster manifest to the kubernetes objects with talosctl, i.e. without omni?

Thanks!

[CompPhy]

@vitali-ipquants I had the same question, but figured it out. Maybe there's a better way, but running talosctl upgrade-k8s seems to do the trick. Just make sure to pass the same kubernetes version already running, so it's not actually doing any kubernetes upgrade.