[Bug]: sky job launch fails with FAILED_PRECHECKS on SSH infra when API server is used, but sky exec succeeds.

[sledress]

Description

When running a SkyPilot managed job (sky job launch) against an SSH cluster using the SkyPilot API Server, the job fails during prechecks with an error suggesting the required SSH support is missing.

However, running the same workload by first provisioning the cluster (sky launch) and then executing the task (sky exec) works successfully.

Steps to Reproduce

1. API Server Configuration (~/.sky/config.yaml excerpt):

   The user is part of the tuam-site workspace where SSH is not disabled.

   tuam-site:
     private: true
     allowed_users:
       - samuel.le-dressay@domain.com
       - ian.clancy@domain.com
     ssh:
       disabled: false # Explicitly enabled
     gcp:
       disabled: true
     aws:
       disabled: true
     kubernetes:
       allowed_contexts:
         - dev_europe-west4_gke-hpc-cp2625-uat

2. Successful Direct Execution (Expected Behavior):

   • Launch: sky launch -c SSH-TEST --infra ssh/* --cpus 0.1+ --memory 0.1+
   • Execute: sky exec SSH-TEST hello-world.yaml
   • Result: Succeeds.

3. Failing Managed Job Execution via API Server (Actual Behavior):

   • Command: sky job launch -c testing-ssh hello-world.yaml
   • Result: Fails with status FAILED_PRECHECKS.

Job file

resources:
  # Use only kubernetes resources
  infra: ssh/*
  cpus: 0.1+
  memory: 0.1+

run: |
  echo "Hello, SkyPilot!"

Error Log

ERROR:root:exec: process returned 127. /usr/local/bin/sky-kube-exec-wrapper: line 3: exec: ssh-tunnel.sh: not found
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read:None, redirect:None, status:None)) after connection broken by  'NewConnectionError('\<urllib3.connection.HTTPSConnection object at 0x77bf39162d40\>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods?limit=1
... (Connection Refused Warnings) ...
E 11-19 15:53:18 recovery\_strategy.py:515] Failure happened before provisioning. Failover reasons: sky.exceptions.ResourcesUnavailableError: Task 'hello-on-ssh' requires SSH Node Pools which is not enabled. To enable access, change the task cloud requirement or run: sky check ssh
E 11-19 15:53:18 controller.py:732] Provision prechecks failed for task 0
E 11-19 15:53:18 controller.py:736] [sky.exceptions.ResourcesUnavailableError] Task 'hello-on-ssh' requires SSH Node Pools which is not enabled. To enable access, change the task cloud requirement or run: sky check ssh
I 11-19 15:53:18 state.py:2267] [sky.exceptions.ResourcesUnavailableError] Task 'hello-on-ssh' requires SSH Node Pools which is not enabled. To enable access, change the task cloud requirement or run: sky check ssh
... (Cleanup logs) ...
✓ Job finished (status: ManagedJobStatus.FAILED\_PRECHECKS).
command terminated with exit code 100

Environment

• SkyPilot Version: 1.0.0.dev20251118 (Running inside a Kubernetes Pod for the API Server)
• Infrastructure: Mixture of GKE and SSH-based servers.
• Observations: The error /usr/local/bin/sky-kube-exec-wrapper: line 3: exec: ssh-tunnel.sh: not found strongly suggests that the container image used for managed job execution (sky job launch) via the API server is missing the necessary ssh-tunnel.sh script or other SSH provisioning tools.
• Tested on all worskspaces including default.

Appendix

sky check ssh
Checking credentials to enable infra for SkyPilot (Workspaces: default, digit-it, apn-lab, tuam-site).

Checking enabled infra for workspace: 'default'
  SSH: enabled [compute]
    SSH Node Pools:
    └── ian-cluster: enabled.

🎉 Enabled infra for workspace: 'default' 🎉
  SSH [compute]
    SSH Node Pools:
    └── ian-cluster

Checking enabled infra for workspace: 'tuam-site'
  SSH: enabled [compute]
    SSH Node Pools:
    └── ian-cluster: enabled.

🎉 Enabled infra for workspace: 'tuam-site' 🎉
  SSH [compute]
    SSH Node Pools:
    └── ian-cluster
Note: The following clouds were disabled because they were not included in allowed_clouds in ~/.sky/config.yaml or disabled for this workspace 'tuam-site': GCP, AWS     

To enable a cloud, follow the hints above and rerun: sky check
If any problems remain, refer to detailed docs at: https://docs.skypilot.co/en/latest/getting-started/installation.html

[kyuds]

Hey @sledress, thank you for raising this! Can you (with sensitive information omitted) give us your kubeconfig for cluster ssh-ian-cluster

[kyuds]

also, could we get the contents of your ~/.sky/ssh_node_pools.yaml file (or any other yaml file you used to create SSH node pool infra)

[sledress]

Hi @kyuds ,

Please find the info below :

The sshNodePools section in our values file.

sshNodePools: |
    ian-cluster:
      user: root
      identity_file: ~/.ssh/id_rsa
      hosts:
        - ip: 10.107.61.176

The current ssh-ian-cluster config

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://127.0.0.1:6443
  name: ssh-ian-cluster
contexts:
- context:
    cluster: ssh-ian-cluster
    user: ssh-ian-cluster
  name: ssh-ian-cluster
current-context: ssh-ian-cluster
kind: Config
preferences: {}
users:
- name: ssh-ian-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --context
      - ssh-ian-cluster
      - --port
      - "6443"
      - --ttl
      - "30"
      - --host
      - 10.107.61.176
      - --user
      - root
      - --ssh-key
      - /root/.ssh/id_rsa
      command: /usr/local/lib/python3.10/site-packages/sky/utils/kubernetes/ssh-tunnel.sh
      env: null
      provideClusterInfo: false

[sledress]

Find also the content of the folder :

root@skypilot-api-server-dddc7d784-wv9fp:/usr/local/lib/python3.10/site-packages/sky/utils/kubernetes# ls -l
total 176
-rw-r--r-- 1 root root     0 Nov 18 12:35 __init__.py
drwxr-xr-x 2 root root  4096 Nov 18 16:55 __pycache__
-rwxr-xr-x 1 root root  2244 Nov 18 12:35 cleanup-tunnel.sh
-rw-r--r-- 1 root root  4754 Nov 18 12:35 config_map_utils.py
-rwxr-xr-x 1 root root  7186 Nov 18 12:35 create_cluster.sh
-rwxr-xr-x 1 root root   984 Nov 18 12:35 delete_cluster.sh
-rw-r--r-- 1 root root 54733 Nov 18 12:35 deploy_remote_cluster.py
-rw-r--r-- 1 root root  2256 Nov 18 12:35 exec_kubeconfig_converter.py
-rw-r--r-- 1 root root  1333 Nov 18 12:35 generate_kind_config.py
-rwxr-xr-x 1 root root 10694 Nov 18 12:35 generate_kubeconfig.sh
-rw-r--r-- 1 root root 10962 Nov 18 12:35 gpu_labeler.py
-rw-r--r-- 1 root root  1224 Nov 18 12:35 k8s_gpu_labeler_job.yaml
-rw-r--r-- 1 root root  3812 Nov 18 12:35 k8s_gpu_labeler_setup.yaml
-rw-r--r-- 1 root root 22142 Nov 18 12:35 kubernetes_deploy_utils.py
-rwxr-xr-x 1 root root  2483 Nov 18 12:35 rsync_helper.sh
-rwxr-xr-x 1 root root 12143 Nov 18 12:35 ssh-tunnel.sh
-rw-r--r-- 1 root root  9046 Nov 18 12:35 ssh_utils.py

[sledress]

FYI,
Retested today with Version: 1.0.0.dev20251119, after a sky ssh down and sky ssh up, and same issue.

[kyuds]

got it. Thank you so much @sledress ! We will try to resolve the issue soon 🙏