ingress-firewall best practice operation

[DrummyFloyd]

Hello,
i've read a lot about this feature (doc + Discussion already present) and i started to try with the --mode=try only on one of my control plane

i've got 3 ControlPlane and 1 worker at the moment on my config

version: v1alpha1
debug: false
persist: true
machine:
  type: controlplane
  token: kh730j.hhc7brlukofrzjn8
  ca:
    crt: redacted
    key: redacted
  certSANs:
    - k8s.drummy.internal
    - 192.168.1.60
    - 127.0.0.1
  kubelet:
    extraArgs:
      cloud-provider: external
      rotate-server-certificates: "true"
    extraMounts:
      - destination: /var/mnt/longhorn
        type: bind
        source: /var/mnt/longhorn
        options:
          - bind
          - rshared
          - rw
    defaultRuntimeSeccompProfileEnabled: true
    disableManifestsDirectory: true
  network:
    hostname: cp-rk1-1
    interfaces:
      - interface: eth0
        dhcp: true
        vip:
          ip: 192.168.1.60
  install:
    disk: /dev/nvme0n1
    extraKernelArgs:
      - net.ifnames=0
    image: factory.talos.dev/metal-installer/6639eb0eea50a47a8b12a57f5e866d4040d68d322b10ce23fa8c928e8c31f52b:v1.11.3
    wipe: false
  features:
    rbac: true
    stableHostname: true
    kubernetesTalosAPIAccess:
      enabled: true
      allowedRoles:
        - os:reader
      allowedKubernetesNamespaces:
        - kube-system
    apidCheckExtKeyUsage: true
    diskQuotaSupport: true
    kubePrism:
      enabled: true
      port: 7445
    hostDNS:
      enabled: true
      forwardKubeDNSToHost: true
  nodeLabels:
    node.kubernetes.io/instance-type: rk1
    topology.kubernetes.io/region: home
    topology.kubernetes.io/zone: home-turingpi
  nodeAnnotations:
    node.longhorn.io/default-node-tags: '["fast","home","turingpi"]'
cluster:
  id: redaced
  secret: redacted
  controlPlane:
    endpoint: https://192.168.1.60:6443
  clusterName: home-lab
  network:
    dnsDomain: cluster.local
    podSubnets:
      - 10.244.0.0/16
    serviceSubnets:
      - 10.96.0.0/12
  token: redacted
  secretboxEncryptionSecret: redacted
  ca:
    crt: redacted
    key: redacted
  aggregatorCA:
    crt: redacted
    key: redacted
  serviceAccount:
    key: redacted
  apiServer:
    extraArgs:
      enable-aggregator-routing: "true"
    certSANs:
      - 192.168.1.60
      - k8s.drummy.internal
      - 192.168.1.60
      - 127.0.0.1
    disablePodSecurityPolicy: true
    auditPolicy:
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
        - level: Metadata
  controllerManager:
    extraArgs:
      bind-address: 0.0.0.0
  proxy:
    extraArgs:
      metrics-bind-address: 0.0.0.0:10249
  scheduler:
    extraArgs:
      bind-address: 0.0.0.0
  discovery:
    enabled: true
    registries:
      kubernetes:
        disabled: true
      service: {}
  etcd:
    ca:
      crt: redacted
      key: redacted
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381
  allowSchedulingOnControlPlanes: true

and the ingress firewall tested

---
apiVersion: v1alpha1
kind: NetworkDefaultActionConfig
ingress: block
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: etcd-ingress
portSelector:
  ports:
    - 2379-2380
  protocol: tcp
ingress:
  - subnet: 192.168.1.61/32
  - subnet: 192.168.1.62/32
  - subnet: 192.168.1.63/32
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: etcd-metrics-ingress
portSelector:
  ports:
    - 2381
  protocol: tcp
ingress:
  - subnet: 192.168.1.61/32
  - subnet: 192.168.1.62/32
  - subnet: 192.168.1.63/32
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: kubelet-ingress
portSelector:
  ports:
    - 10250
  protocol: tcp
ingress:
  - subnet: 10.244.0.0/16
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: apid-ingress
portSelector:
  ports:
    - 50000
  protocol: tcp
ingress:
  - subnet: 0.0.0.0/0
  - subnet: ::/0
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: trustd-ingress
portSelector:
  ports:
    - 50001
  protocol: tcp
ingress:
  - subnet: 0.0.0.0/0
  - subnet: ::/0
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: k8s-api-ingress
portSelector:
  ports:
    - 6443
  protocol: tcp
ingress:
  - subnet: 0.0.0.0/0
  - subnet: ::/0
---
apiVersion: v1alpha1
kind: NetworkRuleConfig
name: cni-vxlan
portSelector:
  ports:
    - 4789
  protocol: tcp
ingress:
  - subnet: 10.244.0.0/16
---

i've follow the best pratice wrote in the documentation, + 1 taht would limit the access to the etcd/metrics servers to only some subnet

but once i tried , see many log error about some 443 endpoint failling to be reached...

Error

E1016 23:31:39.774468       1 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1: Get \"https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" logger="UnhandledError"
W1016 23:31:41.072696       1 logging.go:55] [core] [Channel #8013 SubChannel #8014]grpc: addrConn.createTransport failed to connect to {Addr: "localhost:2379", ServerName: "localhost:2379", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup localhost: operation was canceled"
E1016 23:31:44.148838       1 remote_available_controller.go:462] "Unhandled Error" err="v1alpha1.drummy.cloud failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1alpha1.drummy.cloud\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
E1016 23:31:44.787201       1 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1: Get \"https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" logger="UnhandledError"
E1016 23:31:49.165445       1 remote_available_controller.go:462] "Unhandled Error" err="v1alpha1.drummy.cloud failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1alpha1.drummy.cloud\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
E1016 23:31:49.794051       1 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
E1016 23:31:54.188318       1 remote_available_controller.go:462] "Unhandled Error" err="v1alpha1.drummy.cloud failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1alpha1.drummy.cloud\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
E1016 23:31:54.297850       1 controller.go:102] "Unhandled Error" err=<
	loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp 10.107.148.151:443: i/o timeout
	, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
I1016 23:31:54.297926       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
W1016 23:31:54.297941       1 handler_proxy.go:99] no RequestInfo found in the context
E1016 23:31:54.298066       1 controller.go:146] "Unhandled Error" err=<
	Error updating APIService "v1alpha1.drummy.cloud" with err: failed to download v1alpha1.drummy.cloud: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
	, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
W1016 23:31:54.298111       1 handler_proxy.go:99] no RequestInfo found in the context
E1016 23:31:54.298141       1 controller.go:102] "Unhandled Error" err=<
	loading OpenAPI spec for "v1alpha1.drummy.cloud" failed with: failed to download v1alpha1.drummy.cloud: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
	, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
I1016 23:31:54.299262       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.drummy.cloud: Rate Limited Requeue.
E1016 23:31:54.809486       1 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1: Get \"https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" logger="UnhandledError"
W1016 23:31:55.299827       1 handler_proxy.go:99] no RequestInfo found in the context
E1016 23:31:55.299893       1 controller.go:113] "Unhandled Error" err="loading OpenAPI spec for \"v1alpha1.drummy.cloud\" failed with: Error, could not get list of group versions for APIService" logger="UnhandledError"
I1016 23:31:55.299917       1 controller.go:126] OpenAPI AggregationController: action for item v1alpha1.drummy.cloud: Rate Limited Requeue.
E1016 23:31:59.196628       1 remote_available_controller.go:462] "Unhandled Error" err="v1alpha1.drummy.cloud failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1alpha1.drummy.cloud\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
E1016 23:31:59.809836       1 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1: Get \"https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" logger="UnhandledError"
E1016 23:32:04.197523       1 remote_available_controller.go:462] "Unhandled Error" err="v1alpha1.drummy.cloud failed with: failing or missing response from https://10.110.27.120:443/apis/drummy.cloud/v1alpha1: Get \"https://10.110.27.120:443/apis/drummy.cloud/v1alpha1\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" logger="UnhandledError"
E1016 23:32:04.826664       1 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1: Get \"https://10.107.148.151:443/apis/metrics.k8s.io/v1beta1\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" logger="UnhandledError"
E1016 23:32:04.856464       1 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"

so i had this new manifest

apiVersion: v1alpha1
kind: NetworkRuleConfig
name: https-internal
portSelector:
  ports:
    - 443
  protocol: tcp
ingress:
  - subnet: 10.244.0.0/16

but again got many other error but with different port like 10250 ...

so i don't really understand what i'm doing wrong here, i don't want to try many stuff, and being locked out of my cluster by mistake ..

1. it's because i've set this to only one CP ?
2. should i apply to all CP/worker to make it work ?
3. i missconfigured something ?

any guidance advise to hardening the cluster would be great thank you

[smira]

The documentation provides pretty extensive list of rules.

It's hard to guess what is wrong, but it feels like your rules use 10.244.0.0/16 - is that pod CIDR? or machine IPs?

Please keep in mind that it's host firewall, and most of the time only host addresses should be there.

[DrummyFloyd]

thank for you time !

10.244.0.0/16 it's podSubnet

[smira]

So you should use almost always in the firewall you host cluster addresses, not pod CIDRs.