-
Notifications
You must be signed in to change notification settings - Fork 129
Add NAT64 to enable IPv6 provision in v4 only host #1567
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,6 +12,11 @@ assured that they are persisted. | |
| | Name | Option | Allowed values | Default | | ||
| | :------ | :------- | :--------------- | :-------- | | ||
| | DOCKER_USE_IPV6_INTERNALLY | Choose whether Docker will use IPv6 internally. | "true", "false" | false | | ||
| | ENABLE_NAT64 | Enable NAT64 functionality to emulate IPv6 capable host. | "true", "false" | "false | | ||
| | DNS_UPSTREAM | Specify upstream DNS server for locally run DNS. Only used when ENABLE_NAT64 = "true" | IPv4 or IPv6 address | 10.1.0.2 | | ||
| | DNS64_PREFIX | Prefix for DNS64 server. Only used when ENABLE_NAT64 = "true" | Any valid prefix | 64:ff9b::/96 | | ||
| | LOCAL_DNS_V4 | IPv4 address for local DNS server. Only used when ENABLE_NAT64 = "true". Using loopback is fine, because in the IPv6 setup VMs don't access DNS server over IPv4. | IPv4 address | 127.0.0.2 | | ||
| | LOCAL_DNS_V6 | IPv6 address for local DNS server. Only used when ENABLE_NAT64 = "true". NOT RECOMMENDED TO USE LOOPBACK ::1, because then VMs cannot access the server. | IPv6 address | fd00:abcd::1 | | ||
| | MAX_SURGE_VALUE | This variable defines if controlplane should scale-in or scale-out during upgrade. | 0 (scale-in) or 1 (scale-out) |1| | ||
| | BOOTSTRAP_CLUSTER | Tool for running management/bootstrap cluster. | minikube, kind, tilt | "kind" when using docker as the container runtime (the default on Ubuntu), "minikube" otherwise | | ||
| | IP_STACK | Choose whether the "external" libvirt network will use IPv4, IPv6, or IPv4+IPv6. This network is the primary network interface for the virtual bare metal hosts. Note that this only sets up the underlying network, and fully provisioning IPv6 kubernetes clusters is not yet automated. If IPv6 is enabled, DHCPv6 will be available to the virtual bare metal hosts. | "v4", "v6", "v4v6" (dual-stack)) | v4 | | ||
|
|
@@ -218,6 +223,10 @@ networking, the following additional steps are required: | |
| 1. You need to replace the default IPv4 addresses with IPv6 addresses in the | ||
| environment variables. | ||
|
|
||
| Notice that if your host does not have support for native IPv6, you need to | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "It would most likely be beneficial to support cases where NAT64 and DNS64 are deployed separately, because there could be scenarios where one uses a shared DNS64."
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree. But ATM nobody seems to use IPv6 anyway, so not sure if it makes sense to make this more complicated than needed. |
||
| enable NAT64 and DNS64 to provision the VMs. If you enable NAT64 with the | ||
| variable below, it will also enable DNS64. | ||
|
|
||
| The following variables need to be set: | ||
|
|
||
| ``` sh | ||
|
|
@@ -230,4 +239,5 @@ export EXTERNAL_SUBNET_V6="fd55::/64" | |
| export BARE_METAL_PROVISIONER_SUBNET_IPV6_ONLY=true | ||
| export DOCKER_USE_IPV6_INTERNALLY=true | ||
| export POD_CIDR="fd00:6969::/64" | ||
| export ENABLE_NAT64=true # use if host does not support native IPv6 | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| nat64_action: setup | ||
| ENABLE_NAT64: "{{ lookup('env', 'ENABLE_NAT64') | default('false', true) }}" | ||
| TEMPLATES_PATH: "{{ metal3_dir }}/vm-setup/roles/nat64/files" | ||
| DNS_UPSTREAM: "{{ lookup('env', 'DNS_UPSTREAM') | default('10.1.0.2', true) }}" | ||
| DNS64_PREFIX: "{{ lookup('env', 'DNS64_PREFIX') | default('fd00:ffff:ffff::/96', true) }}" | ||
| # Local DNS is only used in IPv6 only env | ||
| LOCAL_DNS_V4: "{{ lookup('env', 'LOCAL_DNS_V4') | default('127.0.0.2', true) }}" | ||
| LOCAL_DNS_V6: "{{ lookup('env', 'LOCAL_DNS_V6') | default('2001:4860:4860::8888', true) }}" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| - name: Install IPv6 only required packages | ||
| package: | ||
| name: | ||
| - tayga | ||
| - bind9 | ||
| state: present | ||
| when: nat64_action == "setup" | ||
|
|
||
| - name: Setup NAT64 | ||
| block: | ||
| - include_tasks: setup_nat.yml | ||
| when: nat64_action == "setup" | ||
|
|
||
| - name: Teardown NAT64 | ||
| block: | ||
| - include_tasks: teardown_nat.yml | ||
| when: nat64_action == "teardown" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| # Libvirt needs dnsmasq to manage networks. Installing it will likely automatically | ||
| # enable global dnsmasq. We don't want it. | ||
| - name: Ensure dnsmasq is stopped | ||
| ansible.builtin.systemd_service: | ||
| state: stopped | ||
| name: dnsmasq | ||
| become: yes | ||
|
|
||
| - name: Write Tayga configuration | ||
| template: | ||
| src: "../templates/tayga.conf" | ||
| dest: "/etc/tayga.conf" | ||
|
|
||
| - name: ensure IP forwarding is on | ||
| shell: | | ||
| sysctl net.ipv4.ip_forward=1 | ||
| sysctl net.ipv6.conf.all.forwarding=1 | ||
| become: yes | ||
|
|
||
| - name: Start NAT64 service tayga | ||
| ansible.builtin.systemd_service: | ||
| state: started | ||
| name: tayga | ||
| become: yes | ||
|
|
||
| - name: Generate Bind9 options file | ||
| template: | ||
| src: "../templates/named.conf.options" | ||
| dest: /etc/bind/named.conf.options | ||
| owner: root | ||
| group: root | ||
| mode: '0644' | ||
|
|
||
| # IPv6 only supported on ubuntu, so assume that systemd-resolved is | ||
| # the default dns resolution | ||
| - name: Disable systemd-resolved | ||
| ansible.builtin.systemd_service: | ||
| state: stopped | ||
| name: systemd-resolved | ||
| become: yes | ||
|
|
||
| - name: Write /etc/resolv.conf to contain local DNS server | ||
| template: | ||
| src: "../templates/resolv.conf" | ||
| dest: /etc/resolv.conf | ||
| owner: root | ||
| group: root | ||
| mode: '0644' | ||
| backup: true | ||
|
|
||
| # Note that if the address is regular 127.0.0.1, then this will add a duplicate | ||
| # address to the interface but with different subnet mask, the default | ||
| # subnetmask is 8 for IPv4 and | ||
| - name: Add DNS-server IPv4-address to loopback interface | ||
| shell: | ||
| cmd: "ip addr add {{ LOCAL_DNS_V4 }}/32 dev lo" | ||
| become: yes | ||
| ignore_errors: true | ||
|
|
||
| - name: Add DNS-server IPv6-address to loopback interface | ||
| shell: | ||
| cmd: "ip addr add {{ LOCAL_DNS_V6 }}/128 dev lo" | ||
| become: yes | ||
| ignore_errors: true | ||
|
|
||
| - name: Restart Bind9 service | ||
| ansible.builtin.systemd_service: | ||
| state: restarted | ||
| daemon_reload: true | ||
| name: named.service | ||
| become: yes | ||
|
|
||
| - name: Add local hostname to /etc/hosts | ||
| ansible.builtin.lineinfile: | ||
| path: /etc/hosts | ||
| line: 127.0.0.1 {{ ansible_hostname }} |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| - name: Stop NAT64 service tayga | ||
| ansible.builtin.systemd_service: | ||
| state: stopped | ||
| name: tayga | ||
| become: yes | ||
|
|
||
| - name: Clean NAT64 interface | ||
| shell: | | ||
| ip link del nat64 | ||
| become: yes | ||
| ignore_errors: true | ||
|
|
||
| - name: Delete DNS-server IPv4-address from loopback interface | ||
| shell: | ||
| cmd: "ip addr del {{ LOCAL_DNS_V4 }}/32 dev lo" | ||
| become: yes | ||
| ignore_errors: true | ||
|
|
||
| - name: Delete DNS-server IPv6-address from loopback interface | ||
| shell: | ||
| cmd: "ip addr del {{ LOCAL_DNS_V6 }}/128 dev lo" | ||
| become: yes | ||
| ignore_errors: true | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could also add IPv4 and IPv6 addresses to the tunnel device in order to enable sending of ICMP(v6) messages. It is a stupid that the need to be added manually, because they also need to be specified in the tayga configuration file. Well, life is.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, that would be good, but again, I feel that introducing any extra complexity at this point is not really needed. |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there an original setup where we would prefer using CLUSTER_APIENDPOINT_HOST?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is rather annoying. Using host would be usually the way to go, but in this case if we add the brackets around IPv6 address to make it look like host, kubeadm errors and the cluster does not get connection.
Also, host names are not used anywhere, so it should not be a problem to use ATM, but of course moving to use host names later would be annoying.