managed nodes joining IPv6 cluster with IPv4 address

[2tim]

I am trying to add a managed node group to an existing IPv6 eks cluster that is only bottlerocket based nodes. We were able to add the aws linux nodes but had to supply an extra bootstrap arg of --ip-family ipv6 --service-ipv6-cidr ${var.cluster_ipv6_cidr} but no such options exist for bottlerock that I know of. I am also unsure how this even works with bottlerocket. Do I need to ensure that something is enabled for it to join the cluster using an IPv6 address?

[etungsten]

Hi @2tim,

Bottlerocket determines the kubelet node IP based on the IP family of the settings.kubernetes.cluster-dns-ip setting. Normally EKS MNG would bootstrap this value for you in the EC2 instance user-data. Is that not what you're seeing?

If you would like to set the cluster-dns-ip value manually, you can derive it based off of the service IPv6 CIDR by replacing the last hextet with a. E.g. If your cluster has a service IPv6 CIDR of

        "kubernetesNetworkConfig": {
            "serviceIpv6Cidr": "fd3c:540b:7a80::/108",
            "ipFamily": "ipv6"
        },

Then you should set the cluster-dns-ip to fd3c:540b:7a80::a. This should already be done automatically for you by EKS MNG in the EC2 launch template. Can you please check if that's the case?

[2tim]

@etungsten thank you for your help. I was able to set the settings.kubernetes.cluster-dns-ip for our IPv6 clusters. I assume that since our subnet is dualstack that the default for bottlerocket is to set the IPv4 address? I also don't see where the managed node group would be able to set this for you. Maybe this is built into some of the tooling AWS relies on for provisioning managed node groups?

[2tim]

I am using the terraform managed node group module. I don’t think it is setting the cluster-dns-ip. Does this have to be set in the config for the instance?

…

On Fri, Sep 8, 2023 at 4:26 PM Erikson Tung ***@***.***> wrote: Hi @2tim <https://github.com/2tim>, Bottlerocket determines the kubelet node IP based on the IP family of the settings.kubernetes.cluster-dns-ip setting. Normally EKS MNG would bootstrap this value for you in the EC2 instance user-data. Is that not what you're seeing? If you would like to set the cluster-dns-ip value manually, you can derive it based off of the service IPv6 CIDR by replacing the last hextet with a. E.g. If your cluster has a service IPv6 CIDR of "kubernetesNetworkConfig": { "serviceIpv6Cidr": "fd3c:540b:7a80::/108", "ipFamily": "ipv6" }, Then you should set the cluster-dns-ip to fd3c:540b:7a80::a. This should already be done automatically for you by EKS MNG in the EC2 launch template. Can you please check if that's the case? — Reply to this email directly, view it on GitHub <#3435 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHPPPSNEKNZ6E6PMAGTK6DXZOEQNANCNFSM6AAAAAA4Q46LTE> . You are receiving this because you were mentioned.Message ID: <bottlerocket-os/bottlerocket/repo-discussions/3435/comments/6952422@ github.com>

[jpculp]

This is an old one, but I just wanted let folks know that bottlerocket-os/bottlerocket-core-kit#734 improved how the cluster DNS is generated when it hasn't been explicitly set. You'll see the improved behavior starting from Bottlerocket v1.51.0.