Minio reports TLS issue when adding a new pool : tls certificate in secret is not updated

[djeanprost]

Using operator and tenant helm chart 7.1.1, I tried to add a new pool by adding an new entry in tenant helm chart values.yml
The new pod is poped successfuly, but the two tenants fails chatting with each other because of TLS issues :
Error: grid: https://test-minio-pool-0-0.test-minio-hl.test.svc.cluster.local:9000/ re-connecting to https://test-minio-pool-1-0.test-minio-hl.test.svc.cluster.local:9000/: tls: failed to verify certificate: x509: certificate is valid for test-minio-pool-0-0.test-minio-hl.test.svc.cluster.local, minio.test.svc.cluster.local, minio.test, minio.test.svc, *., *.test.svc.cluster.local, not test-minio-pool-1-0.test-minio-hl.test.svc.cluster.local

Expected Behavior

When changing the number of pool (adding or removing), I expect the secret containing the certificates to be recreated to reflect the new alternative names of the new headless service

Current Behavior

For the moment, the secret is not updated, so to force the secret recreation, I must delete it then run the helm upgrade again.

Steps to Reproduce (for bugs)

1. deploy a helm operator and tenant with values with one single pool
2. wait before everything is up
3. then add one pool. Run helm upgrade
4. notice that the tls certificate in secret is not updated, and then there are errors in tenant pods

Your Environment

• Version used (minio-operator): 7.1.1 (problem also occurs with 7.0.0)
• Environment name and version (e.g. kubernetes v1.17.2): kub 1.23

Please notice that this issue is the same than #2307 from my point of view but I suspect @oghoneim managed to fix it accidentaly by running a install from scracth which does work, I can confirm.

[djeanprost]

apiVersion: v1
items:
- apiVersion: minio.min.io/v2
  kind: Tenant
  metadata:
    annotations:
      meta.helm.sh/release-name: minio-tenant
      meta.helm.sh/release-namespace: minio-tenant
      prometheus.io/path: /minio/v2/metrics/cluster
      prometheus.io/port: "9000"
      prometheus.io/scheme: http
      prometheus.io/scrape: "true"
    creationTimestamp: "2025-08-18T13:01:20Z"
    generation: 3
    labels:
      app: minio
      app.kubernetes.io/managed-by: Helm
    name: minio-tenant
    namespace: minio-tenant
    resourceVersion: "313937484"
    uid: 9784bbcf-d622-44a8-b512-d9eb741c1e14
  spec:
    configuration:
      name: myminio-env-custom-configuration
    features:
      bucketDNS: false
      enableSFTP: false
    image: registry-pull.foo.fr/minio/minio:RELEASE.2025-04-08T15-41-24Z
    imagePullPolicy: IfNotPresent
    mountPath: /export
    podManagementPolicy: Parallel
    pools:
    - name: pool-0
      resources:
        limits:
          cpu: "1"
          memory: 1Gi
        requests:
          cpu: "0.5"
          memory: 1Gi
      servers: 1
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
      volumesPerServer: 2
    - name: pool-1
      resources:
        limits:
          cpu: "1"
          memory: 1Gi
        requests:
          cpu: "0.5"
          memory: 1Gi
      servers: 1
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
      volumesPerServer: 2
    poolsMetadata:
      annotations: {}
      labels: {}
    prometheusOperator: false
    requestAutoCert: true
    serviceMetadata:
      consoleServiceAnnotations:
        projectcontour.io/upstream-protocol.tls: https-console
      minioServiceAnnotations:
        projectcontour.io/upstream-protocol.tls: https-minio
      minioServiceLabels:
        monitoring: prometheus
    subPath: /data
  status:
    availableReplicas: 2
    certificates:
      autoCertEnabled: true
      customCertificates: {}
    currentState: Initialized
    drivesOnline: 4
    healthStatus: green
    pools:
    - legacySecurityContext: false
      ssName: minio-tenant-pool-0
      state: PoolInitialized
    - legacySecurityContext: false
      ssName: minio-tenant-pool-1
      state: PoolInitialized
    revision: 0
    syncVersion: v6.0.0
    usage:
      capacity: 9815220224
      rawCapacity: 19630440448
      rawUsage: 14237696
      usage: 7118848
    writeQuorum: 2
kind: List
metadata:
  resourceVersion: ""

[lathil]

Hello,

To add to what Dominique said, I had a look at https://github.com/minio/operator/blob/v7.0.1/pkg/controller/minio.go, in the method checkMinIOCertificatesStatus. From what I understand that is the part of code that check whether to re/generate the tls secret or not, that is used for the pods backing the tenant's pool.

The method only check if the secret for the tls certificate is there or not, or if the expiration date is past. But it does not check whether the certificate alternative names contains all the pools pod internal dns names. In case we add an additional pool to the one existant, this certificate will not be renewed with the new pod internal dns. Hence communication error between the two pods.

Cdt

[ramondeklein]

Which version of the operator are you using? The certificate should also be valid for *.test-minio-hl.test.svc.cluster.local (source), so it would match all the pods that are part of the headless service. Are you specifying a hosts-template to override the default behavior?

[djeanprost]

Which version of the operator are you using?

I'm using version 7.1.1 of operator helm chart

so it would match all the pods that are part of the headless service

in my case, the certificate is available for

[Subject]
  CN=system:node:*.test-minio-hl.test.svc.cluster.local, O=system:nodes

[Issuer]
  CN=kubernetes

[Serial Number]
  73998CC36F830074547DED64C42E43D8

[Not Before]
  18/08/2025 15:28:57

[Not After]
  18/08/2026 15:28:57

[Thumbprint]
  371DE2A3D9A400EF4D7E8C72F2220D3F84072A50

[Utilisation de la clé]
  Signature numérique, Chiffrement de la clé (a0)
  
[Utilisation avancée de la clé]
  Authentification du serveur (1.3.6.1.5.5.7.3.1)
  
[Contraintes de base]
  Type d’objet=Entité finale
  Contrainte de longueur de chemin d’accès=Aucun(e)
  
[Identificateur de clé de l’autorité]
  ID de la clé=22350eeb534c3fb80ee5ad7c0a057c1152f32f57
  
[Autre nom de l’objet]
  Nom DNS=minio-tenant-pool-0-0.test-minio-hl.test.svc.cluster.local
  Nom DNS=minio-tenant-pool-1-0.test-minio-hl.test.svc.cluster.local
  Nom DNS=minio.test.svc.cluster.local
  Nom DNS=minio.minio-tenant
  Nom DNS=minio.test.svc
  Nom DNS=*.
  Nom DNS=*.test.svc.cluster.local

Please notice that the certificate here is a one generated from a 2 pool from-scratch server, this is the reason why you can find the 2 expected alternate DNS name.
When I do the upgrade, the certificate I get contains only one unique alternate name, where the second one is missing.

Are you specifying a hosts-template to override the default behavior?

I don't think so.

[lathil]

Hello Ramon The certificated generated contain a SAN extension with dns entries, so the CN with the wildcard entry is probably rejected by the client. I guess that’w why to work, all pods entries have to be put the the san extension ( or put there the wildcard in the CN .. ) This behaviour is introduced by https://www.rfc-editor.org/rfc/rfc2818.html <https://www.rfc-editor.org/rfc/rfc2818.html> ( deprecated ) and after.

…

Le 19 août 2025 à 15:53, Ramon de Klein ***@***.***> a écrit : ramondeklein left a comment (minio/operator#2485) <#2485 (comment)> Having single node pools is strongly discouraged and probably the reason for this issue. The certificate should have been created for *.test-minio-hl.test.svc.cluster.local, so it would match all the pods that are part of the headless service. Feel free to submit a PR to fix the single-node issue. Single node should only be used for development purposes. — Reply to this email directly, view it on GitHub <#2485 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCE7LR2GTFKAASAAUNRLP33OMT4ZAVCNFSM6AAAAACEHKABR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMBQHA3DGOBYGQ>. You are receiving this because you commented.

[drivebyer]

I have found that AllMinIOHosts() already includes:

"*."+t.MinIOHeadlessServiceHost():

  *.mytenant-hl.default.svc.cluster.local

It should include multiple pools of the same headless service.

[jvansanten]

I ran into a similar issue adding a pool named pool-1 to an existing a tenant named storage. For me, deleting the certificate secret/storage-tls was enough to force the operator to generate a certificate valid for the news pods.

Before:

X509v3 Subject Alternative Name:
    DNS:storage-pool-0-0.storage-hl.lsst-archive.svc.cluster.local, DNS:minio.lsst-archive.svc.cluster.local, DNS:minio.lsst-archive, DNS:minio.lsst-archive.svc, DNS:*., DNS:*.lsst-archive.svc.cluster.local

After:

X509v3 Subject Alternative Name:
    DNS:storage-pool-0-0.storage-hl.lsst-archive.svc.cluster.local, DNS:storage-pool-1-0.storage-hl.lsst-archive.svc.cluster.local, DNS:minio.lsst-archive.svc.cluster.local, DNS:minio.lsst-archive, DNS:minio.lsst-archive.svc, DNS:*., DNS:*.lsst-archive.svc.cluster.local

[jvansanten]

Another possibly related issue: with the default requestAutoCert: true in the manifest, the operator generates certificates with invalid DNS wildcards for multi-node pools, leading to errors like:

hostname verification failed: x509: certificate is valid for storage-pool-1-0.storage-hl.lsst-archive.svc.cluster.local, storage-pool-2-{0...3}.storage-hl.lsst-archive.svc.cluster.local, minio.lsst-archive.svc.cluster.local, minio.lsst-archive, minio.lsst-archive.svc, *., *.lsst-archive.svc.cluster.local, not storage-pool-2-0.storage-hl.lsst-archive.svc.cluster.local

This error comes from x509.Certificate.VerifyHostname(), which can't handle patterns like storage-pool-2-{0...3}.storage-hl.lsst-archive.svc.cluster.local. For me, adding

kind: Tenant
spec:
  # Work around a bug in minio-operator that omits a wildcard DNS entry for the TLS cert
  certConfig:
    dnsNames:
      - "*.storage-hl.lsst-archive.svc.cluster.local"

was enough to generate certificates that are valid for inter-node communication.