plugin fails to connect to apiserver when using IPv6

[endreszabo]

I have problems with the pure default setup of GoCD with all-defaults coming from the Helm chart. The only 'exception' is the Kubernetes namespace name used, which is called gocd-test. Seems like it wants to evaluate the string of fd4d:4045:e5e8:f1::1 against the list of cert SANs. It happens even so when I specify the host via the kubernetes_cluster_url and/or the apiserver signer's cert via kubernetes_cluster_ca_cert.

Static agents (by increased replica count for GoCD Agent ReplicaSets) on the same cluster work. Connection to K8s API server also works from the pod using openssl s_client.

Log for the plugin:

2024-12-16 15:16:47,860 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.
2024-12-16 15:17:47,806 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.
2024-12-16 15:18:07,988 ERROR [195@MessageListener for CreateAgentListener] KubernetesPlugin:127 - Error occurred while trying to list kubernetes pods:
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [gocd-test]  failed.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:453)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:419)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:98)
        at cd.go.contrib.elasticagent.KubernetesAgentInstances.refreshAll(KubernetesAgentInstances.java:156)
        at cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:61)
        at cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:36)
        at cd.go.contrib.elasticagent.executors.CreateAgentRequestExecutor.execute(CreateAgentRequestExecutor.java:55)
        at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:82)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:204)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:163)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131)
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
        at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:59)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:42)
        at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:31)
        at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:21)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Hostname fd4d:4045:e5e8:f1::1 not verified:
    certificate: sha256/Jcl0Z4YxZsf2qBV5kO+qHkKNra+UbdY+Zo+06Pn/C6U=
    DN: CN=kube-apiserver
    subjectAltNames: [fd4d:4045:e5e8:f1:0:0:0:1, fd4d:4045:e5e8:fd:0:0:0:0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, vsh01, vsh01.huszb0.y7.local]
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:451)
        ... 20 common frames omitted
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname fd4d:4045:e5e8:f1::1 not verified:
    certificate: sha256/Jcl0Z4YxZsf2qBV5kO+qHkKNra+UbdY+Zo+06Pn/C6U=
    DN: CN=kube-apiserver
    subjectAltNames: [fd4d:4045:e5e8:f1:0:0:0:1, fd4d:4045:e5e8:fd:0:0:0:0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, vsh01, vsh01.huszb0.y7.local]
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:350)
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        ... 1 common frames omitted
2024-12-16 15:18:07,988 INFO  [195@MessageListener for CreateAgentListener] KubernetesPlugin:72 - Did not find any running kubernetes pods.
2024-12-16 15:18:07,994 INFO  [195@MessageListener for CreateAgentListener] KubernetesPlugin:72 - [Create Agent] Creating K8s pod with spec: Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations={Image=gocd/gocd-agent-wolfi:v24.4.0, PodConfiguration=apiVersion: v1
kind: Pod
metadata:
  name: gocd-agent-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  serviceAccountName: default
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: gocd/gocd-agent-wolfi:v24.4.0, PodSpecType=yaml, Elastic-Agent-Job-Identifier={"pipeline_name":"getting_started_pipeline","pipeline_counter":1,"pipeline_label":"1","stage_name":"default_stage","stage_counter":"1","job_name":"default_job","job_id":1}}, creationTimestamp=2024-12-16T15:18:07.994547100Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={app=web, Elastic-Agent-Created-By=cd.go.contrib.elasticagent.kubernetes, Elastic-Agent-Job-Id=1, kind=kubernetes-elastic-agent}, managedFields=[], name=gocd-agent-25bdb9dc-1d60-427d-8eae-9e591d51db9c, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[], command=[], env=[EnvVar(name=GO_EA_SERVER_URL, value=http://gocd-test-server.gocd-test:8153/go, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_KEY, value=8bec7910-3769-410c-8c83-6a24d0f15220, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_ELASTIC_AGENT_ID, value=gocd-agent-25bdb9dc-1d60-427d-8eae-9e591d51db9c, valueFrom=null, additionalProperties={}), EnvVar(name=GO_EA_AUTO_REGISTER_ELASTIC_PLUGIN_ID, value=cd.go.contrib.elasticagent.kubernetes, valueFrom=null, additionalProperties={})], envFrom=[], image=gocd/gocd-agent-wolfi:v24.4.0, imagePullPolicy=null, lifecycle=null, livenessProbe=null, name=gocd-agent-container-ad0f8367-fbb2-4421-ac86-e87baf874784, ports=[], readinessProbe=null, resizePolicy=[], resources=null, restartPolicy=null, securityContext=null, startupProbe=null, stdin=null, stdinOnce=null, terminationMessagePath=null, terminationMessagePolicy=null, tty=null, volumeDevices=[], volumeMounts=[], workingDir=null, additionalProperties={})], dnsConfig=null, dnsPolicy=null, enableServiceLinks=null, ephemeralContainers=[], hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostUsers=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector={}, os=null, overhead={}, preemptionPolicy=null, priority=null, priorityClassName=null, readinessGates=[], resourceClaims=[], restartPolicy=null, runtimeClassName=null, schedulerName=null, schedulingGates=[], securityContext=null, serviceAccount=null, serviceAccountName=default, setHostnameAsFQDN=null, shareProcessNamespace=null, subdomain=null, terminationGracePeriodSeconds=null, tolerations=[], topologySpreadConstraints=[], volumes=[], additionalProperties={}), status=null, additionalProperties={}).
2024-12-16 15:18:27,128 ERROR [195@MessageListener for CreateAgentListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.create-agent
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
        at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1155)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:98)
        at cd.go.contrib.elasticagent.KubernetesInstanceFactory.createKubernetesPod(KubernetesInstanceFactory.java:151)
        at cd.go.contrib.elasticagent.KubernetesInstanceFactory.createUsingPodYaml(KubernetesInstanceFactory.java:246)
        at cd.go.contrib.elasticagent.KubernetesInstanceFactory.create(KubernetesInstanceFactory.java:59)
        at cd.go.contrib.elasticagent.KubernetesAgentInstances.createKubernetesInstance(KubernetesAgentInstances.java:91)
        at cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:65)
        at cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:36)
        at cd.go.contrib.elasticagent.executors.CreateAgentRequestExecutor.execute(CreateAgentRequestExecutor.java:55)
        at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:82)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:204)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:163)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131)
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
        at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:59)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:42)
        at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:31)
        at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:21)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Hostname fd4d:4045:e5e8:f1::1 not verified:
    certificate: sha256/Jcl0Z4YxZsf2qBV5kO+qHkKNra+UbdY+Zo+06Pn/C6U=
    DN: CN=kube-apiserver
    subjectAltNames: [fd4d:4045:e5e8:f1:0:0:0:1, fd4d:4045:e5e8:fd:0:0:0:0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, vsh01, vsh01.huszb0.y7.local]
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:754)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:98)
        at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42)
        ... 23 common frames omitted
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname fd4d:4045:e5e8:f1::1 not verified:
    certificate: sha256/Jcl0Z4YxZsf2qBV5kO+qHkKNra+UbdY+Zo+06Pn/C6U=
    DN: CN=kube-apiserver
    subjectAltNames: [fd4d:4045:e5e8:f1:0:0:0:1, fd4d:4045:e5e8:fd:0:0:0:0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, vsh01, vsh01.huszb0.y7.local]
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:350)
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        ... 1 common frames omitted
2024-12-16 15:18:47,863 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.
2024-12-16 15:19:47,806 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.

The X.509 attributes and the certificate itself for the API server are:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 7994619493137651346 (0x6ef298125ae21692)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Nov 18 20:32:03 2024 GMT
            Not After : Nov 18 20:37:03 2025 GMT
        Subject: CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:bb:00:aa:6b:cb:a4:85:85:e4:03:0f:54:26:13:
                    9b:84:a8:e2:99:03:64:c7:14:5b:d9:99:19:90:58:
                    83:fd:ce:fb:8e:1e:e3:2e:1f:c2:b1:6c:fa:a9:c6:
                    3d:63:35:e0:97:b3:84:76:b1:0f:42:ec:e8:68:e0:
                    a7:35:c1:a9:4a:20:01:30:df:34:dc:aa:fc:20:10:
                    16:a4:69:a7:5c:8a:55:31:bc:7d:6b:16:02:0a:6f:
                    52:99:c7:10:da:3c:32:78:82:18:c2:3b:01:40:d1:
                    c8:50:f4:bc:41:e0:11:5d:24:e4:98:99:04:2f:73:
                    3b:bb:be:78:3d:18:0d:97:ec:95:ed:4a:86:f1:e9:
                    e2:12:25:42:3c:51:d0:32:a5:3b:a9:89:e0:9b:70:
                    04:df:34:c8:95:2d:34:92:1f:6e:30:e7:e8:2b:6b:
                    b3:6c:3c:03:b7:57:ed:ff:ad:0f:6e:16:d8:68:56:
                    c2:c0:71:55:e7:4e:f3:41:35:4b:9a:04:87:3c:ac:
                    16:ef:e3:c8:f9:8d:32:a7:7c:9e:1b:be:2d:6d:f3:
                    e8:fc:d4:40:fb:66:4c:34:d2:aa:6a:bf:61:84:81:
                    86:a1:b2:61:00:92:7a:66:9b:51:a1:2d:b0:50:74:
                    1b:11:15:b9:56:90:35:ce:49:69:1a:3e:da:fc:8c:
                    27:c5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                16:2B:B8:34:9A:CC:5F:54:42:EA:1C:95:51:6F:0D:21:E7:5C:0B:44
            X509v3 Subject Alternative Name: 
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:vsh01, DNS:vsh01.huszb0.y7.local, IP Address:FD4D:4045:E5E8:F1:0:0:0:1, IP Address:FD4D:4045:E5E8:FD:0:0:0:0
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        37:91:17:78:1d:44:05:74:e7:57:7a:ed:83:d4:8a:f7:b4:a8:
        5f:71:32:4f:28:21:98:09:1c:c1:27:89:25:d1:9b:b9:73:2e:
        52:17:b4:44:6f:4a:1c:98:e4:30:e5:67:d4:9e:7c:79:89:8d:
        e2:8a:3d:6f:19:c8:47:82:08:ba:17:e9:c7:be:9f:66:46:b3:
        02:e6:75:a6:8c:46:54:51:ff:f7:e7:dc:fe:59:11:ad:8f:48:
        1a:1c:3c:b8:99:43:e4:b7:68:96:89:61:1e:92:46:13:0c:d6:
        df:60:51:f8:1f:1f:33:f3:f3:01:ee:ca:4e:b2:74:60:ae:30:
        37:44:2c:af:77:f4:07:45:f0:d8:1b:e7:05:be:99:34:6b:a8:
        91:55:41:c7:9f:ce:1b:a4:0e:e9:fc:07:a5:bb:c6:0e:f6:89:
        f9:1a:b4:95:83:83:d8:65:16:aa:05:28:f6:f2:ca:5f:9d:d3:
        bc:aa:aa:93:04:14:37:34:7d:2e:a9:58:54:be:3b:3c:42:54:
        aa:03:1f:c6:8e:2b:63:21:1d:5b:67:5c:55:54:7e:65:0e:7f:
        93:f3:dc:57:9b:38:33:7d:e1:f0:ae:e0:eb:90:5b:ca:7a:92:
        78:58:b1:10:fa:d9:ea:f3:40:1f:63:93:0c:9e:af:c4:77:bb:
        36:9f:71:19
-----BEGIN CERTIFICATE-----
MIIDuDCCAqCgAwIBAgIIbvKYElriFpIwDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
AxMKa3ViZXJuZXRlczAeFw0yNDExMTgyMDMyMDNaFw0yNTExMTgyMDM3MDNaMBkx
FzAVBgNVBAMTDmt1YmUtYXBpc2VydmVyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEAuwCqa8ukhYXkAw9UJhObhKjimQNkxxRb2ZkZkFiD/c77jh7jLh/C
sWz6qcY9YzXgl7OEdrEPQuzoaOCnNcGpSiABMN803Kr8IBAWpGmnXIpVMbx9axYC
Cm9SmccQ2jwyeIIYwjsBQNHIUPS8QeARXSTkmJkEL3M7u754PRgNl+yV7UqG8eni
EiVCPFHQMqU7qYngm3AE3zTIlS00kh9uMOfoK2uzbDwDt1ft/60PbhbYaFbCwHFV
507zQTVLmgSHPKwW7+PI+Y0yp3yeG74tbfPo/NRA+2ZMNNKqar9hhIGGobJhAJJ6
ZptRoS2wUHQbERW5VpA1zklpGj7a/IwnxQIDAQABo4IBBjCCAQIwDgYDVR0PAQH/
BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0j
BBgwFoAUFiu4NJrMX1RC6hyVUW8NIedcC0QwgasGA1UdEQSBozCBoIIKa3ViZXJu
ZXRlc4ISa3ViZXJuZXRlcy5kZWZhdWx0ghZrdWJlcm5ldGVzLmRlZmF1bHQuc3Zj
giRrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWyCBXZzaDAxghV2
c2gwMS5odXN6YjAueTcubG9jYWyHEP1NQEXl6ADxAAAAAAAAAAGHEP1NQEXl6AD9
AAAAAAAAAAAwDQYJKoZIhvcNAQELBQADggEBADeRF3gdRAV051d67YPUive0qF9x
Mk8oIZgJHMEniSXRm7lzLlIXtERvShyY5DDlZ9SefHmJjeKKPW8ZyEeCCLoX6ce+
n2ZGswLmdaaMRlRR//fn3P5ZEa2PSBocPLiZQ+S3aJaJYR6SRhMM1t9gUfgfHzPz
8wHuyk6ydGCuMDdELK939AdF8Ngb5wW+mTRrqJFVQcefzhukDun8B6W7xg72ifka
tJWDg9hlFqoFKPbyyl+d07yqqpMEFDc0fS6pWFS+OzxCVKoDH8aOK2MhHVtnXFVU
fmUOf5Pz3FebODN94fCu4OuQW8p6knhYsRD62erzQB9jkwyer8R3uzafcRk=
-----END CERTIFICATE-----

[endreszabo]

on the other hand, the plugin status page shows all the details of the cluster just OK

[chadlwilson]

With which values are you configuring the cluster profile when manually configuring? Which plugin version?

[endreszabo]

With which values are you configuring the cluster profile when manually configuring? Which plugin version?

config XML snippet for the plugin:

  <elastic>
    <agentProfiles>
      <agentProfile id="demo-app" clusterProfileId="k8-cluster-profile">
        <property>
          <key>PodSpecType</key>
          <value>yaml</value>
        </property>
        <property>
          <key>Privileged</key>
        </property>
        <property>
          <key>MaxCPU</key>
        </property>
        <property>
          <key>Environment</key>
        </property>
        <property>
          <key>Image</key>
          <value>gocd/gocd-agent-wolfi:v24.4.0</value>
        </property>
        <property>
          <key>PodConfiguration</key>
          <value>apiVersion: v1
kind: Pod
metadata:
  name: gocd-agent-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  serviceAccountName: default
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: gocd/gocd-agent-wolfi:v24.4.0</value>
        </property>
        <property>
          <key>SpecifiedUsingPodConfiguration</key>
        </property>
        <property>
          <key>RemoteFile</key>
        </property>
        <property>
          <key>MaxMemory</key>
        </property>
        <property>
          <key>RemoteFileType</key>
          <value>json</value>
        </property>
      </agentProfile>
      <agentProfile id="testep" clusterProfileId="test">
        <property>
          <key>Image</key>
        </property>
        <property>
          <key>MaxMemory</key>
        </property>
        <property>
          <key>MaxCPU</key>
        </property>
        <property>
          <key>Environment</key>
        </property>
        <property>
          <key>PodConfiguration</key>
          <value>apiVersion: v1
kind: Pod
metadata:
  name: pod-name-prefix-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: {{ GOCD_AGENT_IMAGE }}:{{ LATEST_VERSION }}
      securityContext:
        privileged: true</value>
        </property>
        <property>
          <key>SpecifiedUsingPodConfiguration</key>
        </property>
        <property>
          <key>PodSpecType</key>
          <value>yaml</value>
        </property>
        <property>
          <key>RemoteFile</key>
        </property>
        <property>
          <key>RemoteFileType</key>
          <value>json</value>
        </property>
        <property>
          <key>Privileged</key>
        </property>
      </agentProfile>
    </agentProfiles>
    <clusterProfiles>
      <clusterProfile id="k8-cluster-profile" pluginId="cd.go.contrib.elasticagent.kubernetes">
        <property>
          <key>go_server_url</key>
          <value>http://gocd-test-server.gocd-test:8153/go</value>
        </property>
        <property>
          <key>auto_register_timeout</key>
        </property>
        <property>
          <key>pending_pods_count</key>
        </property>
        <property>
          <key>kubernetes_cluster_url</key>
          <value>kubernetes.default</value>
        </property>
        <property>
          <key>security_token</key>
        </property>
        <property>
          <key>kubernetes_cluster_ca_cert</key>
          <encryptedValue>AES:MV4LVxkmdK/qRcj8MMEFNA==:NceWmoYvcOdToYMm5s/GT3zpsWgqrxX4JRXwtwa1GVCHXs1czhfTmWxTcab5FuSQ6uxdL1eRm07bDaBMQ2xRI43V3t9C5cjSo92o1hUjj7t0BChdIMKq+YTSiaf56yOBWn9B/cAm0xNrAQKZmdAXf6yWhPsFXYK6fgVYELGzImkyX1hHTtf4ZeY3bPVupXSNBJtyogw532REeZecs972EXRI5P7f7jpg3SAm6WwgC2eqlRj/LFPIthvaGvHiUIrR7XclsskMFYnMnm2mRC0Dk6dH75TCzRyPciafXqRfr/F7ZXNhK5gqtMIWibzYCuGn4skFNo9am7f+9XwchvM4kfHaSLgBNxUBaAc+IdSsNvQaFMnr+exCj0LHntC9FPjkhWxLpyMlgbtc8kqKMf4Lnt7vKAU091m8SiwjY/EiJkjapp3muMf2Y/DfG6eE9nSRKuIYY4BXohYQ2Se4yXx+7/BT0uGmToCKWxgt03pWvCcs/6yQmZdOOLotibmDyuAbxCmLc3dHJnXdT9+VZNXUdH4psdQo+EuZgHxsixLUEVHhjSP4j0yGa8Qo1gqhh1R26VO8dg+sWohzJMKqd/S8QDESqxGp3mnSmiuvqtAlYon5gRpMxsylm/ib1PU8JE/ix5NEGYSBtrksm8rSCDro6DlknhosBp7Zncpb6rGcLu+s6HVmRTy7//UF2jaVxy7CdYYGtzBwlK4I+rzB6o55rbcMdyAE1BO2YRtiH8vPtkobyFlr9matfnVaGHa7li6GSUfjN6O8CWvPxM4qwxaTxVnzYMbn4hrbTXttww0tVVzENB3jV9tra3fc8J77f7vLGi7E2eLqJ1JfuMZzGr7x7odcHdL+B8PZF9wHtBErNSXsJfvQTUwk7rRMoN55k1uVJq7X5e9RODNqmKWinDSgm3a3A6egZcvriFVxNsg1y8uk8y4O3S/rUFiYV+vA4QFVwhyIpJHtB+hIZNeGCjNpzkmb3tLZaiY0qY+rSYZmX07Ig5X3ZpTeKcvGhQJd0tbCRfLZzSZc0U6FePT3CmcVJeXQuBNla44RKSTB/7mw61XX1PV2bsxNv9nWEz78EYKWYnQ2ygVB2Ns05Z1PwXU1fOUD2JoZ0ZM8t6Tnt5klYJIqV5eEj1AGpcOh1HOsFX0RiNUYbWHpj8nuoIYSgpzY92tfnZt+yR9r8+E/Vo9WzbxC/PM+jRK3nXfYNQmX3hK5lhWZCT/3zioZgVOf92hft78fPP12lLYfwZlFAjU4ClCgkroTCmzrjCdhu/RVri2nKrxDIM7P6yHqmI4GJctjWJUXUnt1YXBOCa0kMYzfB9JlOCXg8QbEu0+cDzxUXtcgNLXcOxWU9J7pJi47IjmfeZD9zL6/LJSQLZqPDLryKxGez1BuT4k/NLG8VDXbSmNKKKS6F6KNVn4sybXrq4ENqWWDzagS88DLuQTude6nAxDD8yxArR0HjtH5FJRgbGHeRWLzDGj4AEQ+48MSYaiFaA==</encryptedValue>
        </property>
        <property>
          <key>namespace</key>
        </property>
        <property>
          <key>cluster_request_timeout</key>
        </property>
      </clusterProfile>
      <clusterProfile id="test" pluginId="cd.go.contrib.elasticagent.kubernetes">
        <property>
          <key>go_server_url</key>
          <value>http://gocd-test-server.gocd-test:8153/go</value>
        </property>
        <property>
          <key>auto_register_timeout</key>
        </property>
        <property>
          <key>pending_pods_count</key>
        </property>
        <property>
          <key>kubernetes_cluster_url</key>
          <value>kubernetes.default</value>
        </property>
        <property>
          <key>security_token</key>
        </property>
        <property>
          <key>kubernetes_cluster_ca_cert</key>
          <encryptedValue>AES:MV4LVxkmdK/qRcj8MMEFNA==:NceWmoYvcOdToYMm5s/GT3zpsWgqrxX4JRXwtwa1GVCHXs1czhfTmWxTcab5FuSQ6uxdL1eRm07bDaBMQ2xRI43V3t9C5cjSo92o1hUjj7t0BChdIMKq+YTSiaf56yOBWn9B/cAm0xNrAQKZmdAXf6yWhPsFXYK6fgVYELGzImkyX1hHTtf4ZeY3bPVupXSNBJtyogw532REeZecs972EXRI5P7f7jpg3SAm6WwgC2eqlRj/LFPIthvaGvHiUIrR7XclsskMFYnMnm2mRC0Dk6dH75TCzRyPciafXqRfr/F7ZXNhK5gqtMIWibzYCuGn4skFNo9am7f+9XwchvM4kfHaSLgBNxUBaAc+IdSsNvQaFMnr+exCj0LHntC9FPjkhWxLpyMlgbtc8kqKMf4Lnt7vKAU091m8SiwjY/EiJkjapp3muMf2Y/DfG6eE9nSRKuIYY4BXohYQ2Se4yXx+7/BT0uGmToCKWxgt03pWvCcs/6yQmZdOOLotibmDyuAbxCmLc3dHJnXdT9+VZNXUdH4psdQo+EuZgHxsixLUEVHhjSP4j0yGa8Qo1gqhh1R26VO8dg+sWohzJMKqd/S8QDESqxGp3mnSmiuvqtAlYon5gRpMxsylm/ib1PU8JE/ix5NEGYSBtrksm8rSCDro6DlknhosBp7Zncpb6rGcLu+s6HVmRTy7//UF2jaVxy7CdYYGtzBwlK4I+rzB6o55rbcMdyAE1BO2YRtiH8vPtkobyFlr9matfnVaGHa7li6GSUfjN6O8CWvPxM4qwxaTxVnzYMbn4hrbTXttww0tVVzENB3jV9tra3fc8J77f7vLGi7E2eLqJ1JfuMZzGr7x7odcHdL+B8PZF9wHtBErNSXsJfvQTUwk7rRMoN55k1uVJq7X5e9RODNqmKWinDSgm3a3A6egZcvriFVxNsg1y8uk8y4O3S/rUFiYV+vA4QFVwhyIpJHtB+hIZNeGCjNpzkmb3tLZaiY0qY+rSYZmX07Ig5X3ZpTeKcvGhQJd0tbCRfLZzSZc0U6FePT3CmcVJeXQuBNla44RKSTB/7mw61XX1PV2bsxNv9nWEz78EYKWYnQ2ygVB2Ns05Z1PwXU1fOUD2JoZ0ZM8t6Tnt5klYJIqV5eEj1AGpcOh1HOsFX0RiNUYbWHpj8nuoIYSgpzY92tfnZt+yR9r8+E/Vo9WzbxC/PM+jRK3nXfYNQmX3hK5lhWZCT/3zioZgVOf92hft78fPP12lLYfwZlFAjU4ClCgkroTCmzrjCdhu/RVri2nKrxDIM7P6yHqmI4GJctjWJUXUnt1YXBOCa0kMYzfB9JlOCXg8QbEu0+cDzxUXtcgNLXcOxWU9J7pJi47IjmfeZD9zL6/LJSQLZqPDLryKxGez1BuT4k/NLG8VDXbSmNKKKS6F6KNVn4sybXrq4ENqWWDzagS88DLuQTude6nAxDD8yxArR0HjtH5FJRgbGHeRWLzDGj4AEQ+48MSYaiFaA==</encryptedValue>
        </property>
        <property>
          <key>namespace</key>
        </property>
        <property>
          <key>cluster_request_timeout</key>
        </property>
      </clusterProfile>
    </clusterProfiles>
  </elastic>

Plugin Version 4.1.0-606

Funny thing is that I cloned the 'default' profile to a new one called test, and it works!

2024-12-16 15:43:47,814 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.
2024-12-16 15:44:13,935 INFO  [191@MessageListener for CreateAgentListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.
2024-12-16 15:44:13,940 INFO  [191@MessageListener for CreateAgentListener] KubernetesPlugin:72 - [Create Agent] Creating K8s pod with spec: Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations={Image=, MaxM
kind: Pod
metadata:
  name: pod-name-prefix-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: {{ GOCD_AGENT_IMAGE }}:{{ LATEST_VERSION }}
      securityContext:
        privileged: true, SpecifiedUsingPodConfiguration=, PodSpecType=yaml, RemoteFile=, RemoteFileType=json, Privileged=, Elastic-Agent-Job-Identifier={"pipeline_name":"getting_started_pipeline","pipeline_counter"
2024-12-16 15:44:48,613 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 1.
2024-12-16 15:44:49,102 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 1.
2024-12-16 15:45:13,738 INFO  [122@MessageListener for JobStatusListener] KubernetesPlugin:72 - [Job Completion] Terminating elastic agent with id pod-name-prefix-a4d0295f-80c7-43ab-909c-fe4058dc560c on job completi
2024-12-16 15:45:47,813 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.
2024-12-16 15:45:47,815 INFO  [197@MessageListener for ServerPingListener] KubernetesPlugin:72 - [refresh-pod-state] Pod information successfully synced. All(Running/Pending) pod count is 0.

I still have exceptions; I assume it is from the plugin polling the default cluster profile.