Directly support internal minio artifacts in tensorboard

[grudloffev]

Checks

• I have searched the existing issues.
• My request is related to one of the components in the kubeflow/notebooks repository.

Motivation

TL;DR: Currently its not possible to use a kfp artifact (in kubeflow's minio) as object storage target in a tensorboard server in an straightforward manner.

• Current behaviour:

1. Create a tensorboard server with object storage. with an uri like "minio://path/to/artifact"
2. Server creation will be stuck forever (No user facing error whatsoever)
3. Controller logs show that a tb-volume is missing (Really misleading)

This is because we currently only handle gcs or s3 prefixes.

• Expected behaviour:

1. Create a tensorboard server with object storage. with an uri like "minio://path/to/artifact"
2. The controller handles the minio:// prefix and defaults to use the internal minio service url and use the crendentials from mlpipeline-minio-artifact

• Current workaround (Still not tested)

1. Use s3:// prefix in place of minio://
2. Use a podDefault setting the minio server url and credentials

Implementation

No response

Are you willing & able to help?

• I am able to submit a PR!
• I can help test the feature!

[grudloffev]

Tried the workaround:

1. Create PodDefault:

apiVersion: "kubeflow.org/v1alpha1"
kind: PodDefault
metadata:
  name: minio-credentials
  namespace: <user-namespace>
spec:
  selector:
    matchLabels:
      app: "add-minio-credentials"
  desc: "Add minio credentials. Artifact should start with `s3://` instead of `minio://`"
  env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: mlpipeline-minio-artifact
          key: accesskey
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: mlpipeline-minio-artifact
          key: secretkey
    - name: S3_ENDPOINT
      value: minio-service.kubeflow:9000
    - name: S3_USE_HTTPS
      value: "0"
    - name: S3_VERIFY_SSL
      value: "1"
---

2. Create tensorboard server with the option and the minio uri with s3:// in place of minio://

3. No runs are displayed:

Probably need to compile a custom container as outlined here.

[grudloffev]

related issue kubeflow/kubeflow#6493

[grudloffev]

There were some issues in the PodDefault:

1. Using the app label which is already defined in the tensorboard server
2. The value should be set to true

Working PodDefault:

apiVersion: "kubeflow.org/v1alpha1"
kind: PodDefault
metadata:
  name: minio-internal-credentials
  namespace: <user-ns>
spec:
  selector:
    matchLabels:
      s3-config-minio-internal: "true"
  desc: "Add internal minio credentials. Artifact should start with `s3://` instead of `minio://`"
  env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: mlpipeline-minio-artifact
          key: accesskey
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: mlpipeline-minio-artifact
          key: secretkey
    - name: S3_ENDPOINT
      value: minio-service.kubeflow:9000
    - name: S3_USE_HTTPS
      value: "0"
    - name: S3_VERIFY_SSL
      value: "1"

[grudloffev]

With the workaround validated, I think a solution is viable by adding a check for "minio://" prefix in the controller (similarly to how its done for gcs) and conditionally modifying the prefix to s3 and mounting the necesary env variables (as done with the PodDefault)