Skip to content

helm-functions.md

Latest commit

 

History

History
368 lines (283 loc) · 18 KB

helm-functions.md

File metadata and controls

368 lines (283 loc) · 18 KB
Raw

Helm-Based Function Creation

Cloud functions support helm-based functions for orchestration across multiple containers.

Prerequisites

Ensure that your helm charts version does not contain `-` For example `v1` is ok but `v1-test` will cause issues.
  1. The Helm chart must define a mini-service container as the inference entry point.
  2. The name of this service in your helm chart should be supplied by setting helmChartServiceName during the function definition. This allows Cloud Functions to communicate and make inference requests to the "mini-service" endpoint.
The `servicePort` defined within the helm chart should be used as the `inferencePort` supplied during function creation. Otherwise, Cloud Functions will not be able to reach the "mini-service".
  1. Ensure you have pushed your helm chart to your OCI container registry.

Pull Secret Management

All Pod specs in your helm chart will be updated with pull secrets at runtime, so any images are authorized to pull automatically. No other configuration is needed.

Create a Helm-based Function

  1. Ensure your helm chart is uploaded to your registry and adheres to the helm-prereq listed above.

  2. Create the function:

    • Include the following additional parameters in the function definition:

      • helmChart
      • helmChartServiceName

    • The helmChart property should be set to the OCI URL of the helm chart that will deploy the "mini-service". The helm chart URL should follow the format: oci://${REGISTRY}/${REPOSITORY}/charts/$NAME-X.Y.Z.tgz. The chart name should not contain - in the version string.

    • NVCF uses helmChartServiceName for readiness checks and function metrics. It must match the service that exposes the mini-service entry point. Templated service names are not supported.

    • Important: The Helm chart name should not contain underscores or other special symbols, as that may cause issues during deployment.

API example

Please see our sample helm chart used in this example for reference.

Below is an example function creation API call creating a helm-based function:

curl -s -X POST "http://${GATEWAY_ADDR}/v2/nvcf/functions" \
    -H "Host: api.${GATEWAY_ADDR}" \
    -H "Authorization: Bearer $NVCF_TOKEN" \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
    "name": "function_name",
    "inferenceUrl": "v2/models/model_name/versions/model_version/infer",
    "inferencePort": 8001,
    "helmChart": "oci://'${REGISTRY}'/'${REPOSITORY}'/charts/inference-test-1.0.tgz",
    "helmChartServiceName": "service_name",
    "apiBodyFormat": "CUSTOM"
}'
For gRPC-based functions, set `"inferenceURL" : "/gRPC"`. This signals to Cloud Functions that the function is using gRPC protocol and is not expected to have a `/gRPC` endpoint exposed for inferencing requests.
  1. Proceed with function deployment and invocation normally.

Multi-node Helm deployment

To create a multi-node helm deployment, you need to use the following format for the instanceType: <CSP>.GPU.<GPU_NAME>_<number of gpus per node>x[.x<number of nodes>]. For example, DGXC.GPU.L40S_1x is a single L40S instance while ON-PREM.GPU.B200_8x.x2 is two full nodes of 8-way B200.

A sample helm chart for a multi-node deployment can be found in the multi-node helm example.

Multi-node NVLink scheduling

The compute cluster must have the `NVLinkOptimized` attribute. See [NVLink-optimized clusters](./cluster-management/configuration.md#nvlink-optimized-clusters).

NVCF can place the Pods from one multi-node Helm function in a single NVLink GPU clique. Use Gang Scheduling when every Pod must be placed atomically. Use Topology-Aware Scheduling to place that gang in one GPU clique. The guides include examples for direct KAI StatefulSets and workloads managed through Grove and Dynamo. NVCA will create a ComputeDomain to connect GPU workload Pods through IMEX.

Each GPU-enabled Pod must request a full node of GPUs.

Legacy NVCA NVLink partition annotation

NVLink partition placement through these legacy affinity rules is best-effort without KAI Scheduler or Grove topology-aware scheduling. The rules do not provide atomic gang placement. Concurrent Pods can initially land in different cliques, and distinct logical groups can land in the same clique. Use [Gang Scheduling](./cluster-management/gang-scheduling.md) with [Topology-Aware Scheduling](./cluster-management/topology-aware-scheduling.md) when the workload requires all Pods to fit and start in a specific topology domain.

For charts that do not use KAI Scheduler or Grove topology constraints, NVCA supports this legacy Pod template annotation:

spec:
  template:
    metadata:
      annotations:
        dra.nvcf.nvidia.io/required-nvlink-domain-index: "0"

Set the annotation on the Pod template for a Deployment, ReplicaSet, StatefulSet, Job, or CronJob. Set it on metadata.annotations for a standalone Pod. Setting it only on the controller object's metadata does not pass it to the Pods.

The value is a logical partition index within the function. It is not the value of the nvidia.com/gpu.clique node label and does not select a physical rack or clique:

  • Pods with the same value form one logical placement group.
  • Pods with different values get different logical group labels. This does not require the groups to use different GPU cliques.
  • Pods without the annotation share a default logical group.

On an NVLink-optimized cluster, NVCA mutates each admitted Pod as follows:

  • It adds the generated dra.nvcf.nvidia.io/nvlink-domain-partition label. Do not set this label in the chart.
  • For an annotated Pod, it adds required Pod affinity so Pods in the same logical group use one value of the nvidia.com/gpu.clique topology key.
  • For an unannotated Pod, it adds preferred Pod affinity with weight 100. The scheduler can spread these Pods when it cannot satisfy the preference.
  • It requires placement on a node that has the nvidia.com/gpu.clique label.
  • It adds the function's ComputeDomain resource claim to containers that request nvidia.com/gpu, nvidia.com/pgpu, nvidia.com/gpu.shared, or an nvidia.com/mig-* resource.

Limitations

When using Helm Charts to deploy a function, the following limitations need to be taken into consideration.

1. Asset caching

  • For any downloads (such as of assets or models) occurring within your function's containers, download size is limited by the disk space on the node.

2. Inference

Progress/partial response reporting is not supported, including any additional artifacts generated during inferencing. Consider opting for HTTP streaming or gRPC bidirectional support.

3. Security Constraints

Helm charts must conform to certain security standards to be deployable as a function. This means that certain helm and Kubernetes features are restricted in NVCF backends. NVCF will process your helm chart on function creation, then later on deployment with your Helm values and other deployment metadata, to ensure standards are enforced.

NVCF may automatically modify certain objects in your chart so they conform to these standards; it will only do so if modification will not break your chart when it is installed in the targeted backend. Possible areas amenable to modification will be noted in the restrictions section below. Any standard that cannot be enforced by modification will result in error(s) during function creation.

Restrictions

  • Supported k8s artifacts under Helm Chart Namespace are listed below; others will be rejected:

    • ConfigMaps
    • Secrets
    • Services (only type: ClusterIP or none)
    • Deployments
    • ReplicaSets
    • StatefulSets
    • Jobs
    • CronJobs
    • Pods
    • ServiceAccounts
    • Roles
    • Rolebindings
    • PersistentVolumeClaims

  • A rendered Helm chart may contain a maximum of 300 of the aforementioned objects.

  • The only allowed Pod or Pod template volume types are:

    • configMap
    • secret
    • projected.sources.* of any of the above
    • persistentVolumeClaim
    • emptyDir

  • No chart hooks are allowed; if specified in the chart, they will not be executed.

CustomResourceDefinitions in helm charts will be skipped on installation. There is no need to modify your chart to remove them from `helm template` output for NVCF.

Helm charts should conform to these additional security standards. While not enforced now, they will be at a later date.

  • All containers have resource limits for at least cpu and memory (and nvidia.com/gpu, ephemeral-storage if required for certain containers).
  • All Pod's and resources that define a Pod template conform to the Kubernetes Pod Security Standards Baseline and Restricted policies.
  • Pod and container securityContext's conform to these parameters:
    • automountServiceAccountToken must be unset or set to false
    • runAsNonRoot must be explicitly set to true
    • hostIPC, hostPID, and hostNetwork must be unset or set to false
    • No privilege escalation, root capabilities, or non-default Seccomp, AppArmor, or SELinux profiles are allowed. See the Baseline and Restricted Pod security standards for fields that cannot be explicitly set.

Helm Chart Overrides

To override keys in your helm chart values.yml, you can provide the configuration parameter and supply corresponding key-value pairs in JSON format which you would like to be overridden when the function is deployed.

curl -s -X POST "http://${GATEWAY_ADDR}/v2/nvcf/deployments/functions/${FUNCTION_ID}/versions/${FUNCTION_VERSION_ID}" \
 -H "Host: api.${GATEWAY_ADDR}" \
 -H "Authorization: Bearer $NVCF_TOKEN" \
 -H 'accept: application/json' \
 -H 'Content-Type: application/json' \
 -d '{
     "deploymentSpecifications": [{
         "gpu": "L40",
         "backend": "nvcf-default",
         "maxInstances": 2,
         "minInstances": 1,
         "configuration": {
         "key_one": "<value>",
         "key_two": { "key_two_subkey_one": "<value>", "key_two_subkey_two": "<value>" }
     ...
     }]
 }'

Instance health

By default, Cloud Functions considers the health of every Kubernetes object rendered by a Helm chart (for example a Deployment, its active ReplicaSet, and its replica Pods), alongside infrastructure Pods. One unhealthy object can cause the entire instance to be recreated, even when the inference service can still serve traffic. StatusByWorkerReadiness changes this tradeoff by using only the inference worker's readiness to determine instance health.

An instance transitions through these phases:

  • Scheduling: one or more Pods have not yet been assigned to a node.
  • Starting: all Pods are scheduled and NVCA is waiting on readiness.
  • Running: all Helm chart objects and the worker Pod report healthy or ready.

Because health depends on all chart objects, a Pod that is not the inference worker (for example a sidecar Deployment or an init Job) can fail the instance if it enters a terminal state.

Timeouts

The following timeouts govern when a still-unhealthy object fails the instance. The clock for each timeout starts when the described condition is first observed, measured from the Pod launch time unless noted. Values are NVCA defaults. Only Worker Degradation Period is operator-configurable, through the Worker Degradation Period setting in NVCA Configuration.

Timeout Default Cause (when the clock starts) Effect (when exceeded)
Pod scheduling 10 minutes A Pod cannot be scheduled onto a node, for example no node satisfies its resources or affinity. The Pod is marked terminal (PodStuckScheduling) and the instance fails and is re-created. For tasks, the task's max queued duration is used instead of this default.
Image pull error 1 minute A scheduled Pod reports ErrImagePull or ImagePullBackOff on any container or init container. The Pod is marked terminal (ImagePullIssues) and the instance fails. Check registry credentials and image references.
Container restart loop 10 minutes A container or init container restarts at least 3 times. The Pod is marked terminal (stuck initializing, containers in restart loop) and the instance fails.
Init container stuck 2 hours A Pod's init containers have not completed (the Pod is not yet Initialized). The Pod is marked terminal (init container stuck) and the instance fails.
Worker startup 2 hours A running Pod has not become ready during its initial startup, before it has ever reported ready. The Pod is considered degraded and the instance fails.
Worker Degradation Period 30 minutes A Pod that had been ready becomes not ready (containers report not ready) after being initialized. The Pod is considered degraded, the instance is marked degraded, and NVCA kills and re-creates it. Operator-configurable.
Pending timeout (max running) 3 hours Objects remain pending, that is not all objects reach ready, since the instance health condition first went unhealthy. The instance fails with a pending timeout.
Failing objects backoff 90 seconds An object reports a transient failure, for example FailedMount or FailedAttachVolume. NVCA requeues and retries every 30 seconds for up to 90 seconds. If the object is still failing after 90 seconds, the instance fails.

Some conditions fail an instance immediately when default instance health checking is used, without waiting for a timeout:

  • A Pod that enters the Failed phase, including admission rejection (UnexpectedAdmissionError).
  • A Pod with restart policy Never whose non-restartable container terminates with a non-zero exit code is treated as degraded right away, regardless of the Worker Degradation Period.
  • A controller object (Deployment, ReplicaSet, StatefulSet, Job) that reports a terminal condition such as ProgressDeadlineExceeded, a replica-failure condition, or a failed Job.
  • A warning event that indicates a terminal error, such as FailedCreate, FailedUpdate, ReplicaSetCreateError, or a forbidden error, on a tracked object.

StatusByWorkerReadiness will not result in instance failure in any of these scenarios as long as the instance's healthcheck endpoint continues to return a positive status.

Use worker readiness for function health

As described above, any unhealthy object, like a Pod, applied by the Helm chart can fail the function instance. This behavior may not be desirable if some objects in your chart are expected to fail or are not critical to serving inference. When your function is configured with the StatusByWorkerReadiness feature flag, the instance health check performed by NVCF becomes the sole determinant of instance health. The flag is read from the chart at install time to configure the function instance.

Add templates/nvcf-workload-config.yaml to the function chart:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nvcf-workload-config
data:
  config.yaml: |
    featureFlags:
      StatusByWorkerReadiness: true

The ConfigMap must be named nvcf-workload-config and the config.yaml key must exist. Cloud Functions reads this configuration from the chart and does not create the ConfigMap in the instance namespace.

Package and push the chart, then create and deploy the function:

helm package ./my-chart
helm push ./my-chart-1.0.0.tgz \
  oci://${REGISTRY}/${REPOSITORY}/charts

./nvcf-cli function create \
  --name my-helm-function \
  --helm-chart \
    oci://${REGISTRY}/${REPOSITORY}/charts/my-chart-1.0.0.tgz \
  --helm-chart-service my-inference-service \
  --inference-url /infer \
  --inference-port 8000 \
  --health-uri /health/ready \
  --health-port 8000

./nvcf-cli function deploy create \
  --function-id <id> \
  --version-id <id> \
  --instance-type NCP.GPU.H100_1x \
  --gpu H100 \
  --min-instances 1 \
  --max-instances 1

The function reaches RUNNING when the instance health endpoint /health/ready returns HTTP 200, indicating readiness. With this flag enabled:

  • Other chart objects' statuses, including the inference Deployment and Service, are informational only. Their statuses are sent in logs but do not gate RUNNING.
  • If a non-worker Pod goes down while the instance health endpoint reports ready, the instance remains RUNNING. Cloud Functions reports the unhealthy object's status for debugging, and Kubernetes can replace it.
  • If the health endpoint begins to not report ready after the instance has entered the RUNNING state, the instance is marked degraded until it reports ready again or 30 minutes have passed (Worker Degradation Period, see NVCA Configuration), after which the instance is killed and re-created by NVCF. (Note: initial startup behavior with a 2 hour timeout remains the same as default instance health behavior)

Without the ConfigMap, or with the flag set to false, Cloud Functions uses standard health behavior.

See also the optional statusByWorkerReadiness value in examples/function-samples/helmchart-samples/inference-test-sample.