Change in root kubernetes secret causes operator lockout.

[alekc]

I am using a following tenant config

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: minio-home:minio.min.io/Tenant:minio-home/minio-home
    prometheus.io/path: /minio/v2/metrics/cluster
    prometheus.io/port: '9000'
    prometheus.io/scheme: http
    prometheus.io/scrape: 'true'
  labels:
    app: minio
  name: minio-home
  namespace: minio-home
spec:
  configuration:
    name: root-config
  features:
    bucketDNS: false
    enableSFTP: false
  image: quay.io/minio/minio:RELEASE.2025-04-22T22-12-26Z
  imagePullPolicy: IfNotPresent
  mountPath: /export
  podManagementPolicy: Parallel
  pools:
    - name: pool0      
      servers: 1
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 100Gi
      volumesPerServer: 1
  poolsMetadata:
    annotations: {}
    labels: {}
  prometheusOperator: false
  requestAutoCert: true
  subPath: /data

Secret root-config is generated by the external-secret-operator, so far so good. Recently, the secret contents has been rotated, and this caused operator to be locked out with error of wrong auth

Expected Behavior

In case of change (i.e. rotation) of root config secret on minio it should update the root passowrd on the tenant.

Current Behavior

It doesn't, and since it didn't save last known valid value anywhere, its locked out from the tenant.

Possible Solution

Upon succesful connection, save the last known username & password somewhere, potentially in a new secret. Keep that info as fallback in case it's impossible to connect.

Once the drift is observed, old values can be used to connect to the instance and change the root password.

Steps to Reproduce (for bugs)

See above

Context

This affects both rotation and some unexpected changes (i.e. by operator) to the secret. The expectation is that operator should operate.

[ron1]

When the MINIO_ROOT_PASSWORD in the configuration secret config.env is changed, I believe the current expectation is that the tenant statefulset pods will be manually restarted immediately thereafter so that the new credential can take effect. Is this your understanding? If so, is this issue a request to improve upon this current behavior? Note: I currently use eso with password generator to manage the configuration secret and stakater-reloader to restart the sts pods when the secret changes.

[drivebyer]

I believe this issue is related to operational friendliness. When a user updates the associated secret, the operator picks up the new credentials, but the tenant continues using the old password, which leads to the problem. To improve operational friendliness, we could, for example, automatically restart the related pods after the secret is updated, ensuring that the MinIO server uses the latest password. What do you think? @ramondeklein @jiuker

[ramondeklein]

@drivebyer You're actually right that the root cause is that operator already uses the updated credentials, so it cannot restart MinIO using the mc admin restart command. The command is not accepted, because MinIO doesn't use these credentials yet (it needs a restart for that). So, it's kind of a chicken/egg problem. It is possible to restart the pods in case mc admin restart fails, but this will cause down-time (especially on systems that have a lot of MinIO nodes). That's why we don't restart the pods automatically.

Root credentials shouldn't be used for normal use, so we recommend not to change them.

[ramondeklein]

PS: It's hard to detect that the MinIO server environment variables are out-of-sync, because it redacts all the sensitive values.