Unable to create token for CNI kubeconfig error: cni-installer/install.go 478: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443

[kishorech816]

I am trying to install calico CNI over Kamaji Hosted control plane on AWS EKS exposed TCP API server over nginx ingress controller, worker node able to join successfully using kubeadm join as well but calico cni pods are getting crash looped due error: cni-installer/install.go 478: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443.

below are the Kamaji TCP SVC and Endpoint details.

k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 19h

k get ep
NAME ENDPOINTS AGE
kubernetes 172.20.239.123:6443 19h

k get ing -n kamaji-member2
NAME CLASS HOSTS ADDRESS PORTS AGE
member2 nginx member2.example.com xxxxxxxxelb.us-east-1.amazonaws.com 80 19h

k get nodes
NAME STATUS ROLES AGE VERSION
ip-10-23-29-122.ec2.internal NotReady 143m v1.30.11

[kishorech816]

there's a similar issue #331 but it seems closed without any solution.

[prometherion]

10.96.0.1 is the kubernetes.default.svc endpoint and it's managed by kube-proxy.

I would suggest you debug the node and check kube-proxy container logs and the iptables output.

[joseluisgonzalezca]

Apart from that, you have to check that 172.20.239.123:6443 is reachable from your worker node.

[kishorech816]

@joseluisgonzalezca its not reachable outside cluster as its POD CIDR
serviceCidr: 10.96.0.0/16
podCidr: 172.20.0.0/16
TCP API server exposed over ingress so I am assuming any communication from worker nodes to go through API server load balancer.

[prometherion]

TCP API server exposed over ingress so I am assuming any communication from worker nodes to go through API server load balancer.

It doesn't work that way, you always need an IP since kubernetes.default.svc will be resolved at the L3, Ingress is L7.

As CLASTIX we solved this with an Enterprise Addon

[kishorech816]

@prometherion below are the kube-proxy logs .. it seems some misconfig with kube-proxy: "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort connections will be accepted on all local IPs. Consider using --nodeport-addresses primary"

kube-proxy logs:

I0325 11:17:51.397322       1 server.go:677] "Successfully retrieved node IP(s)" IPs=["10.23.29.122"]
I0325 11:17:51.399179       1 conntrack.go:121] "Set sysctl" entry="net/netfilter/nf_conntrack_max" value=131072
I0325 11:17:51.399289       1 conntrack.go:60] "Setting nf_conntrack_max" nfConntrackMax=131072
I0325 11:17:51.399379       1 conntrack.go:121] "Set sysctl" entry="net/netfilter/nf_conntrack_tcp_timeout_established" value=86400
I0325 11:17:51.399437       1 conntrack.go:121] "Set sysctl" entry="net/netfilter/nf_conntrack_tcp_timeout_close_wait" value=3600
E0325 11:17:51.399475       1 server.go:234] "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort connections will be accepted on all local IPs. Consider using `--nodeport-addresses primary`"
I0325 11:17:51.429087       1 server.go:243] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
I0325 11:17:51.429139       1 server_linux.go:483] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR specified for primary IP family" ipFamily="IPv4" clusterCIDRs=null
I0325 11:17:51.429177       1 server_linux.go:169] "Using iptables Proxier"
I0325 11:17:51.436327       1 proxier.go:255] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses" ipFamily="IPv4"
I0325 11:17:51.436804       1 server.go:483] "Version info" version="v1.31.2"
I0325 11:17:51.436887       1 server.go:485] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0325 11:17:51.444404       1 config.go:199] "Starting service config controller"
I0325 11:17:51.444777       1 shared_informer.go:313] Waiting for caches to sync for service config
I0325 11:17:51.444902       1 config.go:105] "Starting endpoint slice config controller"
I0325 11:17:51.444914       1 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
I0325 11:17:51.445028       1 config.go:328] "Starting node config controller"
I0325 11:17:51.445078       1 shared_informer.go:313] Waiting for caches to sync for node config
I0325 11:17:51.548327       1 shared_informer.go:320] Caches are synced for node config
I0325 11:17:51.548371       1 shared_informer.go:320] Caches are synced for service config
I0325 11:17:51.548327       1 shared_informer.go:320] Caches are synced for endpoint slice config

[prometherion]

This is not the root cause of the issue you're facing.

As suggested by @joseluisgonzalezca, ensure your worker nodes can interact with the advertised IP address, as well as ensuring Security Groups are permissive.

[kishorech816]

@prometherion this where I am getting confused why can't worker nodes use api server ip instead of kubenetes default svc ip. advertised IP address means endpoint ip or service ip?

[prometherion]

Worker nodes can join the cluster, as you're doing, with the FQDN/L7.

However, kubernetes.default.svc is always routed to an L3 endpoint, such as an IP which matches the --advertised-address value defined on the Kubernetes API Server.

That's Kubernetes networking foundation, leveraging on host routing (such as L7) required some mangling and changes which are simplified by adopting the Enterprise Addon for the Ingress support.

[kishorech816]

understood and I have remove ingress and exposed API Server through load balancer and now its working as TCP Endpoint able reach from worker nodes.
Now facing issue with streaming logs from kubectl and its probably related to some security group issues.