Merge branch 'main' into patch-1

Author: enoodle

CHANGELOG.md

@@ -6,6 +6,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [v0.10.0] - 20250-11-18
+
 ### Added
 - Added parent reference to SubGroup struct in PodGroup CRD to create a hierarchical SubGroup structure
 - Added the option to configure the names of the webhook configuration resources.
@@ -14,7 +16,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - Added enforcement of the `nvidia` runtime class for GPU pods, with the option to enforce a custom runtime class, or disable enforcement entirely.
 - Added a preferred podAntiAffinity term by default for all services, can be set to required instead by setting `global.requireDefaultPodAffinityTerm`
 - Added support for service-level affinities
-- Added time aware scheduling configurations in scheduling shard
+- Added [time aware scheduling](docs/timeaware/README.md) capabilities
+- Added option to specify container name and type for fraction containers
 
 ### Fixed
 - (Openshift only) - High CPU usage for the operator pod due to continues reconciles

cmd/time-aware-simulator/examples/plot_simple.py

@@ -9,6 +9,8 @@
 parser = argparse.ArgumentParser(description='Plot simulation results from CSV file')
 parser.add_argument('input', nargs='?', default='simulation_results.csv',
                     help='Path to the CSV file (default: simulation_results.csv)')
+parser.add_argument('--output', '-o', type=str, default=None,
+                    help='Save plot to PNG file instead of displaying it')
 args = parser.parse_args()
 
 df = pd.read_csv(args.input)
@@ -38,5 +40,10 @@
 ax2.grid(True, alpha=0.3)
 
 plt.tight_layout()
-plt.show()
+
+if args.output:
+    plt.savefig(args.output, dpi=300, bbox_inches='tight')
+    print(f"Plot saved to {args.output}")
+else:
+    plt.show()
 

deployments/kai-scheduler/templates/rbac/operator.yaml

@@ -43,24 +43,42 @@ rules:
   - validatingwebhookconfigurations
   verbs:
   - create
-  - delete
   - get
   - list
+  - watch
+- apiGroups:
+  - admissionregistration.k8s.io
+  resourceNames:
+  - kai-podgroup-validation-v2alpha2
+  - kai-queue-validation-v2
+  - mutating-kai-admission
+  - validating-kai-admission
+  resources:
+  - mutatingwebhookconfigurations
+  - validatingwebhookconfigurations
+  verbs:
+  - delete
   - patch
   - update
-  - watch
 - apiGroups:
   - apiextensions.k8s.io
   resources:
   - customresourcedefinitions
   verbs:
   - create
-  - delete
   - get
   - list
+  - watch
+- apiGroups:
+  - apiextensions.k8s.io
+  resourceNames:
+  - queues.scheduling.run.ai
+  resources:
+  - customresourcedefinitions
+  verbs:
+  - delete
   - patch
   - update
-  - watch
 - apiGroups:
   - apps
   resources:

docs/batch/batch-job.yaml

@@ -11,7 +11,7 @@ spec:
   template:
     metadata:
       labels:
-        kai.scheduler/queue: test
+        kai.scheduler/queue: default-queue
     spec:
       schedulerName: kai-scheduler
       restartPolicy: OnFailure

docs/batch/pytorch-job.yaml

@@ -6,7 +6,7 @@ kind: "PyTorchJob"
 metadata:
   name: "pytorch-dist-mnist-nccl"
   labels:
-    kai.scheduler/queue: test
+    kai.scheduler/queue: default-queue
 spec:
   pytorchReplicaSpecs:
     Master:

docs/developer/designs/time-aware-fairness/time-aware-fairness.md

@@ -116,7 +116,7 @@ Where:
 - **$C$** is the remaining capacity (max amount to give in current round)
 - **$P'_i$** is the normalized portion for queue i, defined as:
 
-$$P_i = \max{\{W'_i - k \cdot (W'_i - U'_i), 0\}}$$
+$$P_i = \max{\{W'_i + k \cdot (W'_i - U'_i), 0\}}$$
 
 $$P'_i = \frac{P_i}{\sum{P}}$$
 

docs/dra/gpu-imex-pod.yaml

@@ -16,7 +16,7 @@ kind: Pod
 metadata:
   name: gpu-imex-pod
   labels:
-    kai.scheduler/queue: test
+    kai.scheduler/queue: default-queue
 spec:
   schedulerName: kai-scheduler
   containers:

docs/elastic/pytorch-elastic.yaml

@@ -6,7 +6,7 @@ kind: PyTorchJob
 metadata:
   name: elastic-example-imagenet
   labels:
-    kai.scheduler/queue: test
+    kai.scheduler/queue: default-queue
 spec:
   elasticPolicy:
     rdzvBackend: c10d

docs/gpu-sharing/README.md

@@ -42,3 +42,31 @@ kubectl apply -f gpu-memory.yaml
 In the gpu-memory.yaml file, the pod includes a `gpu-memory` annotation with a value of 2000 (in Mib), meaning:
 * The pod is allowed to consume up to 2000 Mib of a GPU device memory
 * The remaining GPU device memory can be shared with other pods in the cluster
+
+### GPU Fraction with Non-Default Container
+By default, GPU fraction allocation is applied to the first container (index 0) in the pod. However, you can specify a different container to receive the GPU allocation using the `gpu-fraction-container-name` annotation.
+
+#### Specific Container
+To allocate GPU fraction to a specific container in a multi-container pod:
+```
+kubectl apply -f gpu-sharing-non-default-container.yaml
+```
+
+In the gpu-sharing-non-default-container.yaml file, the pod includes:
+* `gpu-fraction: "0.5"` - Requests half of a GPU device memory
+* `gpu-fraction-container-name: "gpu-workload"` - Specifies that the container named "gpu-workload" should receive the GPU allocation instead of the default first container
+
+This is useful for pods with sidecar containers where only one specific container needs GPU access.
+
+#### Init Container
+To allocate GPU fraction to an init container:
+```
+kubectl apply -f gpu-sharing-init-container.yaml
+```
+
+In the gpu-sharing-init-container.yaml file, the pod includes:
+* `gpu-fraction: "0.5"` - Requests half of a GPU device memory
+* `gpu-fraction-container-name: "gpu-init"` - Specifies the init container name. If not defined, will default to the first container.
+* `gpu-fraction-container-type: "InitContainer"` - Indicates the container is an init container
+
+This is useful for workloads that need GPU access during initialization (e.g., model loading, dataset preprocessing) before the main application container starts.

docs/gpu-sharing/gpu-memory.yaml

@@ -6,12 +6,13 @@ kind: Pod
 metadata:
   name: gpu-sharing
   labels:
-    kai.scheduler/queue: test
+    kai.scheduler/queue: default-queue
   annotations:
     gpu-memory: "2000" # in Mib
 spec:
   schedulerName: kai-scheduler
   containers:
-    - name: ubuntu
-      image: ubuntu
-      args: ["sleep", "infinity"]
+    - name: gpu-workload
+      image: nvidia/cuda:13.0.2-base-ubi8
+      command: ["nvidia-smi"]
+      args: ["-L"]