Adding top level useDualstackEndpoint configuration #254

Paramadon · 2025-12-01T21:27:21Z

Issue #, if available

Currently customers can not use our default configuration to have agent and fluent bit use dualstack endpoints. We want to add a top level field to allow using dualstack endpoints.

Passing tests: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/19836119811

Description of changes:

If customer sets the new top level useDualstackEndpoint field to true, we'll add use_dualstack_endpoint to true in the agent configuration so that it uses dualstack endpoints(this is the current way to enable cloudwatch agent to use dualstack endpoints.)

As for fluent bit, we'll update endpoints/sts_endpoint for fluent bit configuration (under [OUTPUT] section) to use dualstack endpoints and then have fluent bit prefer ipv6 dns using net.dns.prefer_ipv6 true. (Note setting useDualstackEndpoint to true on an ipv4 instance would NOT break the cluster as dualstack endpoints also work on ipv4 instances and net dns prefer ipv6 does fall back to ipv4. (Applied helm on ipv4 instance and set useDualstackEndpoint, and it works.

Also we do aren't replacing endpoints if there is already an endpoint field (for example adc regions). ADC regions are out of scope of this change.

Testing

All testing was done on IPv6 EKS cluster, we also tested that dualstack endpoints work on K8s so we can make this helm chart change.

Below are logs from fluent bit and the agent configuration showcasing what happens if we set the top level field useDualstackEndpoint to true or false.

Setting `useDualstackEndpoint` to true

Helm upgrade command:

helm upgrade --install amazon-cloudwatch-observability \
  charts/amazon-cloudwatch-observability \
  -n amazon-cloudwatch \
  -f charts/amazon-cloudwatch-observability/values.yaml \
  --set clusterName=<cluster-name> \
  --set region=us-west-2 \
  --set useDualstackEndpoint=true

Resulting agent configuration

 config: '{"agent":{"region":"us-west-2","use_dualstack_endpoint":true},"logs":{"metrics_collected":{"application_signals":{"hosted_in":"Ipv6-auto-mode-6n"},"kubernetes":{"cluster_name":"Ipv6-auto-mode-6n","enhanced_container_insights":true}}},"traces":{"traces_collected":{"application_signals":{}}}}

As you can setting useDualstackEndpoint field to true sets agent configuration to use dualstack endpoint.

Resulting fluent bit config

[SERVICE]
 net.dns.prefer_ipv6       true
 Flush                     5
 Grace                     30
 Log_Level                 error
 Daemon                    off
 Parsers_File              parsers.conf
 storage.path              /var/fluent-bit/state/flb-storage/
 storage.sync              normal
 storage.checksum          off
 storage.backlog.mem_limit 5M
 
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker

[OUTPUT]
 Name                cloudwatch_logs
 Match               application.*
 region              ${AWS_REGION}
 endpoint            logs.${AWS_REGION}.api.aws
 sts_endpoint        sts.${AWS_REGION}.api.aws
 log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
 log_stream_prefix   ${HOST_NAME}-
 auto_create_group   true
 extra_user_agent    container-insights

Resulting fluent bit logs

As for Fluent bit, here are some log output showcasing successfully sending logs using dualstack endpoints:

Setting `useDualstackEndpoint` to false

Helm upgrade command:

helm upgrade --install amazon-cloudwatch-observability \
  charts/amazon-cloudwatch-observability \
  -n amazon-cloudwatch \
  -f charts/amazon-cloudwatch-observability/values.yaml \
  --set clusterName=<cluster-name> \
  --set region=us-west-2 \
  --set useDualstackEndpoint=false

Resulting agent configuration

config: '{"agent":{"region":"us-west-2"},"logs":{"metrics_collected":{"application_signals":{"hosted_in":"Ipv6-auto-mode-6n"},"kubernetes":{"cluster_name":"Ipv6-auto-mode-6n","enhanced_container_insights":true}}},"traces":{"traces_collected":{"application_signals":{}}}}

As you can see seeting useDualstackEndpoint to false does not set the use_dualstack_endpoint to true nor adds it to the agent configuration.

Resulting fluent bit config

[SERVICE]
 Flush                     5
 Grace                     30
 Log_Level                 error
 Daemon                    off
 Parsers_File              parsers.conf
 storage.path              /var/fluent-bit/state/flb-storage/
 storage.sync              normal
 storage.checksum          off
 storage.backlog.mem_limit 5M
 
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker


[OUTPUT]
 Name                cloudwatch_logs
 Match               application.*
 region              ${AWS_REGION}
 log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
 log_stream_prefix   ${HOST_NAME}-
 auto_create_group   true
 extra_user_agent    container-insights

Resulting fluent bit logs

As for Fluent bit, here are some log output showcasing successfully sending logs using regular ipv4 endpoints.

Test to ensure we don't override endpoint if it is already given

In the fluent bit configuration I added endpoint logs.us-east-1.api.aws to see if the endpoint is overwritten with useDualstackEndpoint is true and it isn't overridden and the fluent bit configuration does take precedence making sure that we don't override adc region endpoints.

Added scenario tests

Added minikube-based integration tests to validate the useDualstackEndpoint configuration:

What each test validates:

default_test.go: Confirms FluentBit configs don't contain dualstack endpoints or IPv6 preference when disabled

dualstack_endpoint_enabled_test.go: Confirms FluentBit OUTPUT sections have endpoint: logs.${AWS_REGION}.api.aws and sts_endpoint: sts.${AWS_REGION}.api.aws, SERVICE section has net.dns.prefer_ipv6 true, and CloudWatch Agent has use_dualstack_endpoint: true

dualstack_endpoint_with_custom_endpoint_test.go: Confirms custom endpoints in application-log.conf are NOT overwritten, while other configs still get dualstack endpoints

…eplace adc endpoints)

Paramadon · 2025-12-01T21:28:44Z

charts/amazon-cloudwatch-observability/templates/_helpers.tpl

+*/}}
+{{- define "fluent-bit.add-dualstack-endpoints" -}}
+{{- $config := .config -}}
+{{- if and .Values.useDualstackEndpoint (not (contains "endpoint" $config)) -}}


This is to make sure we do not replace endpoint if it already exist in the config (for example for adc regions). ADC is out of scope for this change.

do we have a test that validates this logic?

Added to pr description, endpoint is not overwritten if it already exists in [OUTPUT] section of fluent bit configuration. So adc region endpoint override would be respected.

it would be helpful to add before/after FB config with ipv6 enabled similar to how you show agent config in the description

Paramadon · 2025-12-01T21:29:45Z

charts/amazon-cloudwatch-observability/templates/_helpers.tpl

 {{- $agent := set $configCopy "agent" $agentRegion }}
 {{- end }}

+{{- if .Values.useDualstackEndpoint }}


We're adding the agent section if does not exist and then adding "use_dualstack_endpoint" to true in that config.

Paramadon · 2025-12-01T23:23:19Z

charts/amazon-cloudwatch-observability/templates/linux/fluent-bit-configmap.yaml

    k8s-app: fluent-bit
 data:
  fluent-bit.conf: |
+    {{- if .Values.useDualstackEndpoint }}


We'll set dns prefer ipv6 to true even if environment is ipv4 it should still works as it will fallback to ipv4 as shown by this passing test:
https://github.com/aws/amazon-cloudwatch-agent/actions/runs/19839229481/job/56845933521

Ran test by enabling dualstack endpoint for ipv4 and ipv6 and they both pass.

the-mann

please add manual test to description that tests the case where the endpoint is manually set and dualstackendpoint is enabled. the test should validate that the custom endpoint that is set is used instead of the dual stack endpoint. #254 (comment)

duhminick · 2025-12-03T16:04:13Z

It would be worth adding some test scenarios in https://github.com/aws-observability/helm-charts/tree/main/integration-tests/amazon-cloudwatch-observability/terraform/minikube/scenarios for the logic in here imo

duhminick · 2025-12-03T19:38:50Z

charts/amazon-cloudwatch-observability/templates/_helpers.tpl

+{{- define "fluent-bit.add-dualstack-endpoints" -}}
+{{- $config := .config -}}
+{{- if and .Values.useDualstackEndpoint (not (contains "endpoint" $config)) -}}
+{{- $config = replace "region              ${AWS_REGION}" (printf "region              ${AWS_REGION}\n  endpoint            logs.${AWS_REGION}.api.aws\n  sts_endpoint        sts.${AWS_REGION}.api.aws") $config -}}


Hmm, this seems like it could break at any point depending on the spacing. I think we could use a regexReplaceAll or something

yup that makes sense, using regex now instead.

movence · 2025-12-03T21:45:28Z

charts/amazon-cloudwatch-observability/templates/_helpers.tpl

+*/}}
+{{- define "fluent-bit.add-dualstack-endpoints" -}}
+{{- $config := .config -}}
+{{- if and .Values.useDualstackEndpoint (not (contains "endpoint" $config)) -}}


it would be helpful to add before/after FB config with ipv6 enabled similar to how you show agent config in the description

movence · 2025-12-03T22:08:39Z

charts/amazon-cloudwatch-observability/templates/windows/fluent-bit-windows-configmap.yaml

    {{- range $key, $val := .Values.containerLogs.fluentBit.configWindows.extraFiles }}
    @INCLUDE {{ $key }}
    {{- end }}
  parsers.conf: |


has this been also tested on an windows instance?

removing this altogether as windows nodes don't support ipv6 yet so there is no need for dualstack endpoints: https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html

movence · 2025-12-03T22:13:00Z

charts/amazon-cloudwatch-observability/templates/_helpers.tpl

+{{- define "fluent-bit.add-dualstack-endpoints" -}}
+{{- $config := .config -}}
+{{- if and .Values.useDualstackEndpoint (not (contains "endpoint" $config)) -}}
+{{- $config = mustRegexReplaceAll "(region\\s+\\$\\{AWS_REGION\\})" $config "$1\n  endpoint            logs.$${AWS_REGION}.api.aws\n  sts_endpoint        sts.$${AWS_REGION}.api.aws" -}}


I think custom endpoint logic also applies to sts_endpoint? if so, please add or update minukube test case to include sts_endpoint

movence · 2025-12-03T22:31:36Z

charts/amazon-cloudwatch-observability/templates/linux/fluent-bit-configmap.yaml

    {{- end }}
  parsers.conf: |
    {{- .Values.containerLogs.fluentBit.config.customParsers  | nindent 4 }}
 {{- if hasPrefix "us-iso-" .Values.region }}


not an expert with FB config but what happens if a custom fluentbit config doesn't have Parsers_File for some reasons? would it be better to just appned right after [SERVICE]?

So custom fluent bit configs are out of scope as that's what we also putting in public docs, but it does make sense to append right after [SERVICE], will change it to this instead.

what do you mean custom config is out of scope? the public doc should have some instructions on how to setup ipv6 with custom configs, right? The new function fluent-bit.add-ipv6-preference will skip if a custom config already has dual stack configred.

Yup we are providing customers using custom configuration of a way to update the fluent bit configuration, meant to say that if customer are using customer config they would need to update their fluent bit config manually. This top level field is only for customer using default config.

...n-cloudwatch-observability/validations/minikube/scenarios/dualstack_endpoint_enabled_test.go

Paramadon added 2 commits November 25, 2025 13:12

Adding top level useDualstackEndpoint configuration

cf4fdfb

ensure we only add endpoint if it doesn't already exist(so we don't r…

8b1f13c

…eplace adc endpoints)

Paramadon commented Dec 1, 2025

View reviewed changes

This was referenced Dec 2, 2025

Adding IPv6 e2e tests to test workflow aws/amazon-cloudwatch-agent#1950

Merged

Adding IPv6 Integration Test Infrastructure aws/amazon-cloudwatch-agent-test#624

Merged

the-mann requested changes Dec 2, 2025

View reviewed changes

the-mann previously approved these changes Dec 2, 2025

View reviewed changes

Adding dualstack endpoint test scenario

c2689c7

Paramadon dismissed the-mann’s stale review via 06e2f2c December 3, 2025 19:02

Paramadon force-pushed the paramadon/IPv6EKSTest branch 2 times, most recently from 579fc89 to 811fdac Compare December 3, 2025 19:18

duhminick reviewed Dec 3, 2025

View reviewed changes

Paramadon force-pushed the paramadon/IPv6EKSTest branch 2 times, most recently from 7fe19c1 to fe58e20 Compare December 3, 2025 20:02

Adding fluent bit tests

8d305bb

Paramadon force-pushed the paramadon/IPv6EKSTest branch from fe58e20 to 8d305bb Compare December 3, 2025 20:14

Paramadon requested review from duhminick and the-mann December 3, 2025 21:58

movence reviewed Dec 3, 2025

View reviewed changes

fixing dualstack endpoints

cb4f9ff

Paramadon requested a review from movence December 4, 2025 13:37

duhminick reviewed Dec 4, 2025

View reviewed changes

...n-cloudwatch-observability/validations/minikube/scenarios/dualstack_endpoint_enabled_test.go Show resolved Hide resolved

Paramadon requested a review from duhminick December 5, 2025 16:02

Paramadon force-pushed the paramadon/IPv6EKSTest branch from 9456e18 to c152d57 Compare December 5, 2025 16:16

Removing dupe func

96a6fcb

Paramadon force-pushed the paramadon/IPv6EKSTest branch from c152d57 to 96a6fcb Compare December 5, 2025 16:27

duhminick approved these changes Dec 5, 2025

View reviewed changes

movence approved these changes Dec 5, 2025

View reviewed changes

Paramadon merged commit 3b66fd4 into main Dec 5, 2025
19 checks passed

Paramadon deleted the paramadon/IPv6EKSTest branch December 5, 2025 20:06

Adding top level useDualstackEndpoint configuration #254

Adding top level useDualstackEndpoint configuration #254

Uh oh!

Conversation

Paramadon commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue #, if available

Description of changes:

Testing

Setting useDualstackEndpoint to true

Helm upgrade command:

Resulting agent configuration

Resulting fluent bit config

Resulting fluent bit logs

Setting useDualstackEndpoint to false

Helm upgrade command:

Resulting agent configuration

Resulting fluent bit config

Resulting fluent bit logs

Test to ensure we don't override endpoint if it is already given

Added scenario tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

the-mann left a comment

Choose a reason for hiding this comment

Uh oh!

duhminick commented Dec 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Paramadon Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Paramadon commented Dec 1, 2025 •

edited

Loading

Setting `useDualstackEndpoint` to true

Setting `useDualstackEndpoint` to false

Paramadon Dec 4, 2025 •

edited

Loading