RBAC Issue? : Standard 'Users' cannot view job logs in SkyPilot dashboard when using API Server

[sledress]

Description

We are deploying the SkyPilot API Server on a Kubernetes cluster (GKE/EKS) for team use and are leveraging the built-in Authentication and RBAC features to manage user permissions.

The Problem

Users assigned to the standard User role can successfully:

1. Log in to the SkyPilot dashboard.
2. See the list of their submitted jobs.
3. View basic job metadata.

However, when they attempt to view the detailed job logs for their own runs via the dashboard, the logs section remains empty.

Admin users, correctly see the logs.

Request

We have reviewed the RBAC documentation but have not found explicit instructions detailing which specific Kubernetes permissions need to be granted to the User role to enable log viewing for their own jobs.

Could you please provide guidance on, or update the documentation with, the precise Kubernetes RBAC rules (Resource, Verb, etc.) required for a SkyPilot User to access their job logs in the dashboard?

[DanielZhangQD]

Hi @sledress, thanks for raising this issue!

1. The RBAC rules do not control the endpoint for jobs logs, here are the endpoints that are checked.
2. I verified the case where a user with User role can check the logs of the jobs with an API server with SSO enabled.

Could you please help confirm the following questions for further information?

1. Do you use SSO for the authentication of your API server? Refer to the doc for more details.
2. For the issue above, it is for the same managed job that an Admin user can see the logs but the User users cannot?
3. Could you please provide the output of sky api info?
4. Could you please share your ~/.sky/config.yaml on your API server?
5. Is it OK that you share your task.yaml used to launch the managed job?
6. Do you see any errors in the Network requests or the console in the browser? You can check this with F12 (for Chrome), and refresh the page, an example:

I will use the above info to further debug this issue.

Thanks!

[sledress]

Hello @DanielZhangQD ,

• Yes we do use the SSO and it works perfectly fine. We see user names in the dashboards, etc...
• Yes. And I tested by giving him the admin rights, and reconnect, then he was able to see the logs.
• Our config yaml

kubernetes:
  provision_timeout: 30
  autoscaler: gke
  dws:
    enabled: false
  allowed_contexts:
    - dev_europe-west4_gke-hpc-cp2625-sdbx
    - dev_europe-west4_gke-hpc-cp2625-uat
  context_configs:
    dev_europe-west4_gke-hpc-cp2625-sdbx:
      autoscaler: gke
    dev_europe-west4_gke-hpc-cp2625-uat:
      autoscaler: gke
  pod_config:
    spec:
      imagePullSecrets:
        - name: jfrog-secret
logs:
  store: gcp
  gcp:
    project_id: cp2625-dev
gcp:
  vpc_name: mlops-v3-hpc
  prioritize_reservations: true
jobs:
  controller:
    resources:
      infra: k8s/*
      cpus: 1+
      memory: 1+
      autostop:
        idle_minutes: 5
        down: false
workspaces:
  default:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-sdbx
        - dev_europe-west4_gke-hpc-cp2625-uat
  digit-it:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-uat
        - dev_europe-west4_gke-hpc-cp2625-sdbx
  apn-lab:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
      - chandan@domain.com
    ssh:
      disabled: false
      allowed_node_pools:
        - apn-lab-cluster
    gcp:
      disabled: false
    aws:
      disabled: true
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-uat
        - dev_europe-west4_gke-hpc-cp2625-sdbx
  tuam-site:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
    ssh:
      disabled: false
      allowed_node_pools:
        - ian-cluster
    gcp:
      disabled: true
    aws:
      disabled: true
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-uat
  creteil-site:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
    ssh:
      disabled: true
    gcp:
      disabled: false
    aws:
      disabled: true
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-uat

For the other info, I ask the tester to provide you with.

[sledress]

Here are the Network Issues

Getting 500, with user rights, when it is ok, with admin rights.

[DanielZhangQD]

Hi @sledress, thanks for providing the info!

Could you please help collect more info?

1. The output of sky api info?
2. sky api logs $request_id, where request_id is the id that returns 500 above.

Thanks!

[DanielZhangQD]

Hi @sledress, could you please help double-check the user (email) that launched the job matches with the user who views the job logs?
From the above info, it seems that the email that launched the managed job does not match the email that is configured in the allowed_users for the target workspace.

[shivakumar-18]

Hi @DanielZhangQD , Thank you for the quick and insightful guidance! I have collected the requested information,

1. Output of sky api info

The API server health and user authentication are successful:

2. Output of sky api logs $request_id

I used the request_id (948030b3-8460-41c9-99e0-49f26be4aa34) obtained from the F12 inspection:

$ sky api logs 948030b3-8460-41c9-99e0-49f26be4aa34
sky.exceptions.PermissionDeniedError: User shiva-kumar.kavati@domain.com (e10788b5) does not have permission to access workspace 'default'

[DanielZhangQD]

Hi @shivakumar-18, thanks for the info!
The permission issue is the cause, you need to ask the Admin user to add shiva-kumar.kavati@domain.com to the allowed_users of the default namespace or any other private namespaces that this user needs to access:

workspaces:
  default:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com

[sledress]

Hi @DanielZhangQD ,

In fact @shivakumar-18 is member of the apn-lab team, I just removed all user emails from the config.yaml file when I shared it with you for more privacy 😄

Here is the workspace section again, with all info a little bit scrambled.

workspaces:
  default:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
      - sebastien.belot@domain.com
      - ian.clancy@domain.com
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-sdbx
        - dev_europe-west4_gke-hpc-cp2625-uat
  digit-it:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
      - sebastien.belot@domain.com
      - hanady.abdelbaqi@domain.com
      - abdelrhman.awida@domain.com
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-uat
        - dev_europe-west4_gke-hpc-cp2625-sdbx
  apn-lab:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
      - shiva-kumar.kavati@domain.com
      - chandan.venkataswamy-reddy@domain.com
      - abdelrhman.awida@domain.com
    ssh:
      disabled: false
      allowed_node_pools:
        - apn-lab-cluster
    gcp:
      disabled: false
    aws:
      disabled: true
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-uat
        - dev_europe-west4_gke-hpc-cp2625-sdbx
  tuam-site:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
      - ian.clancy@domain.com
      - shiva-kumar.kavati@domain.com
    ssh:
      disabled: false
      allowed_node_pools:
        - ian-cluster
    gcp:
      disabled: true
    aws:
      disabled: true
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-uat
  creteil-site:
    private: true
    allowed_users:
      - samuel.le-dressay@domain.com
      - adrien.loizeau@domain.com
      - christel.leonet@domain.com
      - saif-eddine.boukhdhir@domain.com
    ssh:
      disabled: true
    gcp:
      disabled: false
    aws:
      disabled: true
    kubernetes:
      allowed_contexts:
        - dev_europe-west4_gke-hpc-cp2625-uat

[DanielZhangQD]

Hi @sledress, the issue is caused by the default workspace set to private, with the current configuration, the default active workspace on the API server is the default workspace, while for the requests from the dashboard, no active workspace config is set, so the final active workspace for the requests from the dashboard is the default workspace, in this case, the requests for the users without permission to the default namespace will fail.

Is it OK to set default namespace as public and not allow users to use it with the following config?

  default:
    private: false
    kubernetes:
      disabled: true
    ssh:
      disabled: true
    aws:
      disabled: true
    gcp:
      disabled: true