feat(operator): add strict ordering of Skyhooks using priority and name

chart/templates/skyhook-crd.yaml

@@ -421,12 +421,6 @@ spec:
                   type: object
                 description: Packages are the DAG of packages to be applied to nodes.
                 type: object
-              pause:
-                default: false
-                description: |-
-                  Pause halt the operator from proceeding. THIS is for admin use to stop skyhook if there is an issue or
-                  concert without needing to delete to ad in discovery of the issue.
-                type: boolean
               podNonInterruptLabels:
                 description: PodNonInterruptLabels are a set of labels we want to
                   monitor pods for whether they Interruptible
@@ -479,6 +473,11 @@ spec:
                 description: This skyhook is required to have been completed before
                   any workloads can start
                 type: boolean
+              priority:
+                description: Priority determines the order in which skyhooks are applied. Lower values are applied first.
+                type: integer
+                minimum: 0
+                default: 200
               serial:
                 default: false
                 description: Serial tells skyhook if it allowed to run in parallel or

docs/ordering_of_skyhooks.md

@@ -0,0 +1,24 @@
+# Ordering of Skyhooks
+## What
+With v0.8.0 Skyhooks now always get applied in a repeatable and specific order. This also means that all Skyhooks will now be sequential, though packages within a Skyhook can be parallel. Each custom resource now supports a `priority` field which is a non-zero positive integer. Skyhooks will be processed in order starting from 0, any Skyhooks with the same `priority` will be processed by sorting them by their `metadata.name` field.
+
+**NOTE**: Any Skyhook which does NOT provide a `priority` field will be assigned a priority value of 200.
+
+Two additional flow control features have been added with this and can be set in the annotations of each skyhook:
+ * `skyhook.nvidia.com/disable`: bool. When `true` it will skip this Skyhook from processing and continue with any other ones further down the priority order.
+ * `skyhook.nvidia.com/pause`: bool. When `true` it will NOT process this Skyhook and it WILL NOT continue to process any Skyhook's after this one. This will effectively stop all application of Skyhooks starting with this one. NOTE: This ability used to be on the Skyhook spec itself as the `pause` field and has been moved here to be consistent with `disable` and to avoid incrementing the generation of a Skyhook Custom Resource instance when changing it.
+
+## Why
+This solves a few problems:
+
+The first is to to better support debugging. Prior to this it was impossible to know the order Skyhooks would get applied to nodes as they would all run in parallel. This can, and has, lead to issues debugging a problem as it isn't deterministic. Now every node will always receive updates in the same order as every other node. Additionaly, this removes the possiblility of conflicts between Skyhooks by heaving each one run in order.
+
+The second is to provide the ability for complex tasks to be sequenced. This comes up when needing to apply different sets of work to different node groups in a particular order.
+
+The third is to provide the community a way to bucket Skyhooks according to where they might live in a stream of updates and therefore better coordinate work without explicit communication. We propose the following buckets:
+ * 1 - 99 for initialization and infrastucture work
+    * install security or monitoring tools
+ * 100 - 199 for configuration work
+    * configuring ssh access
+ * 200+ for final user level configuration
+    * applying tuning for workloads

examples/interrupt-wait-for-pod/scr.yaml

@@ -25,6 +25,9 @@ metadata:
     app.kubernetes.io/part-of: skyhook-operator
     app.kubernetes.io/created-by: skyhook-operator
   name: demo
+  annotations:
+    skyhook.nvidia.com/pause: "false"
+    skyhook.nvidia.com/disable: "false"
 spec:
   nodeSelectors:
     matchLabels:

examples/simple/scr.yaml

@@ -25,17 +25,23 @@ metadata:
     app.kubernetes.io/part-of: skyhook-operator
     app.kubernetes.io/created-by: skyhook-operator
   name: demo
+  annotations:
+    skyhook.nvidia.com/pause: "false"
+    skyhook.nvidia.com/disable: "false"
 spec:
+  nodeSelectors:
+    matchLabels:
+      skyhook.nvidia.com/test-node: skyhooke2e
   packages:
     baz:
       version: 1.1.0
       image: ghcr.io/nvidia/skyhook-packages/shellscript
       configMap:
         config.yaml: |-
           #!/bin/bash
-          sleep 30
+          sleep 1
           echo "Hello, config!"
         config_check.yaml: |-
           #!/bin/bash
-          sleep 30
+          sleep 1
           echo "Hello, config check!"

k8s-tests/chainsaw/skyhook/cleanup-pods/chainsaw-test.yaml

@@ -24,7 +24,6 @@ kind: Test
 metadata:
   name: cleanup-pods
 spec:
-  concurrent: true 
   description: |
     This test runs a simple skyhook with dependsOn. We wait tell completed, then trigger update to force config cycle on package B. Once config
     is complete, we update again to make the package error, and at the same clear out the node annotation to trigger cleanup.

k8s-tests/chainsaw/skyhook/config-skyhook/chainsaw-test.yaml

@@ -31,7 +31,6 @@ spec:
     accordingly but the package with the config changes may be skipeed and hang, and this asserts that it doesn't hang in that condition. Then once that completes the same two
     packages have the key with a config interrupt changed and it's asserted that the config, interrupt, and post-interrupt runs for both those packages. Then once that completes 
     again it does one more update on a key for the same two packages which doesn't have a config interrupt defined and makes sure that the config steps are ran for that.
-  concurrent: true
   timeouts:
     assert: 360s
   catch: ## if errors, print the most important info

k8s-tests/chainsaw/skyhook/config-skyhook/skyhook.yaml

@@ -38,6 +38,9 @@ spec:
       interrupt: 
         type: service
         services: [cron]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
     dexter:
       version: "1.2.3"
       image: ghcr.io/nvidia/skyhook/agentless
@@ -47,6 +50,9 @@ spec:
         game.properties:
           type: service
           services: [rsyslog]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
       configMap:
         game.properties: |
           enemies=aliens
@@ -65,6 +71,9 @@ spec:
         game.properties:
           type: service
           services: [rsyslog, cron]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
       configMap:  
         game.properties: |
           enemies=aliens

k8s-tests/chainsaw/skyhook/config-skyhook/update-no-interrupt.yaml

@@ -38,6 +38,9 @@ spec:
       interrupt: 
         type: service
         services: [cron]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
     dexter:
       version: "1.2.3"
       image: ghcr.io/nvidia/skyhook/agentless
@@ -47,6 +50,9 @@ spec:
         game.properties:
           type: service
           services: [rsyslog]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
       configMap:
         game.properties: |
           changed
@@ -59,6 +65,9 @@ spec:
         game.properties:
           type: service
           services: [rsyslog, cron]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
       configMap:  
         game.properties: |
           changed again

k8s-tests/chainsaw/skyhook/config-skyhook/update-while-running.yaml

@@ -38,6 +38,9 @@ spec:
       interrupt: 
         type: service
         services: [cron]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
     dexter:
       version: "1.2.3"
       image: ghcr.io/nvidia/skyhook/agentless
@@ -47,6 +50,9 @@ spec:
         game.properties:
           type: service
           services: [rsyslog]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
       configMap:
         game.properties: |
           enemies=aliens
@@ -61,6 +67,9 @@ spec:
     baxter:
       version: "3.2.1"
       image: ghcr.io/nvidia/skyhook/agentless
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
       configInterrupts:
         game.properties:
           type: service

k8s-tests/chainsaw/skyhook/config-skyhook/update.yaml

@@ -38,6 +38,9 @@ spec:
       interrupt: 
         type: service
         services: [cron]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
     dexter:
       version: "1.2.3"
       image: ghcr.io/nvidia/skyhook/agentless
@@ -47,6 +50,9 @@ spec:
         game.properties:
           type: service
           services: [rsyslog]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
       configMap:
         game.properties: |
           changed
@@ -62,6 +68,9 @@ spec:
         game.properties:
           type: service
           services: [rsyslog, cron]
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
       configMap:  
         game.properties: |
           changed again

k8s-tests/chainsaw/skyhook/depends-on/chainsaw-test.yaml

@@ -24,7 +24,6 @@ kind: Test
 metadata:
   name: depends-on
 spec:
-  concurrent: true 
   description: |
     Test makes sure depends-on works as expected. c depends on a, and b. Make sure a and b complete before c starts.
   timeouts:

k8s-tests/chainsaw/skyhook/failure-skyhook/chainsaw-test.yaml

@@ -24,7 +24,6 @@ kind: Test
 metadata:
   name: failure-skyhook
 spec:
-  concurrent: true 
   timeouts:
     assert: 240s
   catch: ## if errors, print the most important info

k8s-tests/chainsaw/skyhook/failure-skyhook/skyhook.yaml

@@ -38,6 +38,8 @@ spec:
       env: 
         - name: EXIT_CODE
           value: "2"
+        - name: SLEEP_LEN
+          value: "1"
       dependsOn:
         dexter: "1.2.3"
     dexter:

k8s-tests/chainsaw/skyhook/interrupt/chainsaw-test.yaml

@@ -31,7 +31,6 @@ spec:
     Additionally this test tests the agentImageOverride, interruption budgets, and dependsOn fields.
   ## this can't run concurrently because it can cause a race condition where the other skyhooks make the node unschedulable 
   ## and the pods won't come up
-  concurrent: false
   timeouts:
     assert: 240s
   catch: ## if errors, print the most important info

k8s-tests/chainsaw/skyhook/pod-finalizer/chainsaw-test.yaml

@@ -24,7 +24,6 @@ kind: Test
 metadata:
   name: pod-finalization
 spec:
-  concurrent: true
   timeouts:
     assert: 180s
   steps:

k8s-tests/chainsaw/skyhook/runtime-required/chainsaw-test.yaml

@@ -24,7 +24,6 @@ kind: Test
 metadata:
   name: runtime-required
 spec:
-  concurrent: false 
   timeouts:
     assert: 240s
   catch: ## if errors, print the most important info

k8s-tests/chainsaw/skyhook/runtime-required/skyhook.yaml

@@ -34,4 +34,7 @@ spec:
     spencer:
       version: "3.2.3"
       image: ghcr.io/nvidia/skyhook/agentless
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
 

k8s-tests/chainsaw/skyhook/simple-skyhook/chainsaw-test.yaml

@@ -24,7 +24,7 @@ kind: Test
 metadata:
   name: simple-skyhook
 spec:
-  concurrent: false ## because this test creates a limit range in the namespace, it must be run serially
+  # skip: false ##  this test doesn't seem to be useful, just slows things down, leaving it for, should delete at some point if still skipped
   timeouts:
     assert: 240s
   steps:

k8s-tests/chainsaw/skyhook/simple-skyhook/skyhook.yaml

@@ -40,14 +40,17 @@ spec:
       image: ghcr.io/nvidia/skyhook/agentless
       dependsOn:
         dexter: "1.2.3"
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
     foobar:
       version: "1.2"
       image: ghcr.io/nvidia/skyhook/agentless
       dependsOn:
         dexter: "1.2.3"
       env: 
         - name: SLEEP_LEN
-          value: "3" ## making faster so the test works for asserting node condition
+          value: "1" ## making faster so the test works for asserting node condition
       resources:
         cpuLimit: 50m
         cpuRequest: 50m
@@ -67,3 +70,6 @@ spec:
           color.bad=yellow
           allow.textmode=true
           how.nice.to.look=fairlyNice
+      env: 
+        - name: SLEEP_LEN
+          value: "1"

k8s-tests/chainsaw/skyhook/simple-update-skyhook/chainsaw-test.yaml

@@ -24,7 +24,6 @@ kind: Test
 metadata:
   name: simple-update-skyhook
 spec:
-  concurrent: true
   timeouts:
     assert: 240s
   catch: ## if errors, print the most important info

k8s-tests/chainsaw/skyhook/simple-update-skyhook/skyhook.yaml

@@ -37,12 +37,15 @@ spec:
       image: ghcr.io/nvidia/skyhook/agentless
       dependsOn:
         dexter: "1.2.3"
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
     foobar:
       version: "1.2"
       image: ghcr.io/nvidia/skyhook/agentless
       env: 
         - name: SLEEP_LEN
-          value: "3" ## making faster so the test works for asserting node condition
+          value: "1" ## making faster so the test works for asserting node condition
       dependsOn:
         dexter: "1.2.3"
     dexter:
@@ -58,4 +61,7 @@ spec:
           color.good=purple
           color.bad=yellow
           allow.textmode=true
-          how.nice.to.look=fairlyNice
+          how.nice.to.look=fairlyNice
+      env: 
+        - name: SLEEP_LEN
+          value: "1"

k8s-tests/chainsaw/skyhook/simple-update-skyhook/update.yaml

@@ -46,9 +46,12 @@ spec:
           color.bad=yellow
           allow.textmode=true
           how.nice.to.look=fairlyNice
+      env: 
+        - name: SLEEP_LEN
+          value: "1"
     jackie-chan: ## TODO: need to handle bad naming, because this are pod names, so can have issues depending on the chars
       version: "2024.7.7-test"
       image: ghcr.io/nvidia/skyhook/agentless
       env: 
         - name: SLEEP_LEN
-          value: "3" ## making faster so the test works for asserting node condition
+          value: "1" ## making faster so the test works for asserting node condition

k8s-tests/chainsaw/skyhook/skyhook-upgrade/chainsaw-test.yaml

@@ -24,7 +24,6 @@ kind: Test
 metadata:
   name: skyhook-upgrade
 spec:
-  concurrent: true 
   skip: true ## skipping because this test current requires manual updating of the version
   description: |
     This test is skipped because it is because its not automated to change versions while its running.

k8s-tests/chainsaw/skyhook/skyhook-upgrade/skyhook.yaml

@@ -38,4 +38,4 @@ spec:
       image: ghcr.io/nvidia/skyhook/agentless
       env: 
         - name: SLEEP_LEN
-          value: "3" ## making faster so the test works for asserting node condition
+          value: "1" ## making faster so the test works for asserting node condition