Skip to content

fix: race bug running more then one pod at a time #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 26, 2025
Merged

fix: race bug running more then one pod at a time #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions chart/templates/skyhook-crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -431,8 +431,8 @@ spec:
concert without needing to delete to ad in discovery of the issue.
type: boolean
podNonInterruptLabels:
description: PodNonInteruptLabels are a set of labels we want to monitor
pods for whether they Interruptible
description: PodNonInterruptLabels are a set of labels we want to
monitor pods for whether they Interruptible
properties:
matchExpressions:
description: matchExpressions is a list of label selector requirements.
Expand Down
10 changes: 0 additions & 10 deletions k8s-tests/chainsaw/skyhook/interrupt-grouping/assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,9 @@
# LICENSE END
#







kind: Pod
apiVersion: v1
metadata:
generateName: interrupt-grouping-dax-1.2.3-apply-
namespace: skyhook
labels:
skyhook.nvidia.com/name: interrupt-grouping
Expand Down Expand Up @@ -61,7 +54,6 @@ spec:
kind: Pod
apiVersion: v1
metadata:
generateName: interrupt-grouping-dax-1.2.3-config-
namespace: skyhook
labels:
skyhook.nvidia.com/name: interrupt-grouping
Expand Down Expand Up @@ -92,7 +84,6 @@ spec:
kind: Pod
apiVersion: v1
metadata:
generateName: interrupt-grouping-interrupt-service-
namespace: skyhook
labels:
skyhook.nvidia.com/name: interrupt-grouping
Expand Down Expand Up @@ -132,7 +123,6 @@ spec:
kind: Pod
apiVersion: v1
metadata:
generateName: interrupt-grouping-dax-1.2.3-post-interrupt-
namespace: skyhook
labels:
skyhook.nvidia.com/name: interrupt-grouping
Expand Down
9 changes: 0 additions & 9 deletions k8s-tests/chainsaw/skyhook/interrupt/assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,10 @@
# LICENSE END
#







---
kind: Pod
apiVersion: v1
metadata:
generateName: interrupt-invalid-1.2.3-uninstall-
namespace: skyhook
labels:
skyhook.nvidia.com/name: interrupt
Expand Down Expand Up @@ -63,7 +56,6 @@ spec:
kind: Pod
apiVersion: v1
metadata:
generateName: interrupt-jason-1.3.2-apply-
namespace: skyhook
labels:
skyhook.nvidia.com/name: interrupt
Expand Down Expand Up @@ -126,7 +118,6 @@ data:
kind: Pod
apiVersion: v1
metadata:
generateName: interrupt-interrupt-service-
namespace: skyhook
labels:
skyhook.nvidia.com/name: interrupt
Expand Down
10 changes: 2 additions & 8 deletions k8s-tests/chainsaw/skyhook/interrupt/pod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,6 @@
# LICENSE END
#







---
apiVersion: v1
kind: Pod
Expand Down Expand Up @@ -64,8 +58,8 @@ metadata:
name: invalid-package
namespace: skyhook
labels:
skyhook.nvidia.com/package: invalid-1.2.3
skyhook.nvidia.com/name: interrupt
skyhook.nvidia.com/package: invalid-1.2.3
skyhook.nvidia.com/name: interrupt
annotations:
skyhook.nvidia.com/package: '{"name":"invalid","version":"1.2.3","stage":"apply","skyhook":"interrupt","image":"ghcr.io/nvidia/skyhook/agentless"}'
spec:
Expand Down
6 changes: 0 additions & 6 deletions k8s-tests/chainsaw/skyhook/interrupt/skyhook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,6 @@
# LICENSE END
#







---
apiVersion: skyhook.nvidia.com/v1alpha1
kind: Skyhook
Expand Down
7 changes: 0 additions & 7 deletions k8s-tests/chainsaw/skyhook/simple-skyhook/assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,10 @@
# LICENSE END
#







---
kind: Pod
apiVersion: v1
metadata:
generateName: simple-skyhook-spencer-3.2.3-apply-
namespace: skyhook
labels:
skyhook.nvidia.com/name: simple-skyhook
Expand Down
8 changes: 0 additions & 8 deletions k8s-tests/chainsaw/skyhook/uninstall-upgrade-skyhook/assert-update-no-packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,10 @@
# LICENSE END
#







---
kind: Pod
apiVersion: v1
metadata:
generateName: uninstall-upgrade-skyhook-dogs-1.2.5-uninstall-
namespace: skyhook
labels:
skyhook.nvidia.com/name: uninstall-upgrade-skyhook
Expand Down Expand Up @@ -63,7 +56,6 @@ spec:
kind: Pod
apiVersion: v1
metadata:
generateName: uninstall-upgrade-skyhook-nullptr-2.0.1-uninstall-
namespace: skyhook
labels:
skyhook.nvidia.com/name: uninstall-upgrade-skyhook
Expand Down
9 changes: 0 additions & 9 deletions k8s-tests/chainsaw/skyhook/uninstall-upgrade-skyhook/assert-update.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,10 @@
# LICENSE END
#







---
kind: Pod
apiVersion: v1
metadata:
generateName: uninstall-upgrade-skyhook-cats-6.2.0-uninstall-
namespace: skyhook
labels:
skyhook.nvidia.com/name: uninstall-upgrade-skyhook
Expand Down Expand Up @@ -77,7 +70,6 @@ spec:
kind: Pod
apiVersion: v1
metadata:
generateName: uninstall-upgrade-skyhook-dogs-1.2.6-uninstall-
namespace: skyhook
labels:
skyhook.nvidia.com/name: uninstall-upgrade-skyhook
Expand Down Expand Up @@ -112,7 +104,6 @@ spec:
kind: Pod
apiVersion: v1
metadata:
generateName: uninstall-upgrade-skyhook-nullptr-2.0.1-upgrade-
namespace: skyhook
labels:
skyhook.nvidia.com/name: uninstall-upgrade-skyhook
Expand Down
26 changes: 10 additions & 16 deletions k8s-tests/chainsaw/skyhook/validate-packages/assert-update.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,6 @@
# LICENSE END
#







apiVersion: v1
kind: Node
metadata:
Expand Down Expand Up @@ -73,24 +67,24 @@ status:
observedGeneration: 3
nodeState:
(values(@)):
- invalid-image|3.2.3:
name: invalid-image
state: complete
version: '3.2.3'
- invalid-env|5.4.3:
image: ghcr.io/nvidia/skyhook/agentless
name: invalid-env
stage: config
invalid-resources|1.2.3:
name: invalid-resources
state: complete
version: '1.2.3'
version: 5.4.3
invalid-image|3.2.3:
image: ghcr.io/nvidia/skyhook/agentless
name: invalid-image
stage: config
invalid-env|5.4.3:
name: invalid-env
state: complete
version: '5.4.3'
version: 3.2.3
invalid-resources|1.2.3:
image: ghcr.io/nvidia/skyhook/agentless
name: invalid-resources
stage: config
state: complete
version: 1.2.3
nodeStatus:
# grab values should be one and is complete
(values(@)):
Expand Down
4 changes: 2 additions & 2 deletions operator/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ $(REPORTING):
mkdir -p $@

.PHONY: test
test:: reporting manifests generate fmt vet lint helm-tests e2e-tests unit-tests ## Run all tests.
test:: reporting manifests generate fmt vet lint unit-tests e2e-tests helm-tests ## Run all tests.

ifndef CI
## we double define test so we can do thing different if in ci vs local
Expand All @@ -166,7 +166,7 @@ unit-tests: reporting manifests generate envtest ginkgo kill ## Run unit tests.
## https://kyverno.github.io/chainsaw/latest/configuration/file/
CHAINSAW_ARGS:=--exec-timeout 30s --parallel 2

e2e-tests: chainsaw run ## Run end to end tests.
e2e-tests: chainsaw install run ## Run end to end tests.
## requires a cluster to be running with access
## locally use kind to create clusters
## in ci, the plan current is to have a real cluster, and create a node pool for testing
Expand Down
11 changes: 7 additions & 4 deletions operator/api/v1alpha1/skyhook_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ type SkyhookSpec struct {
//+kubebuilder:default=false
Pause bool `json:"pause,omitempty"`

// PodNonInteruptLabels are a set of labels we want to monitor pods for whether they Interruptible
PodNonInteruptLabels metav1.LabelSelector `json:"podNonInterruptLabels,omitempty"`
// PodNonInterruptLabels are a set of labels we want to monitor pods for whether they Interruptible
PodNonInterruptLabels metav1.LabelSelector `json:"podNonInterruptLabels,omitempty"`

// NodeSelector are a set of labels we want to monitor nodes for applying packages too
NodeSelector metav1.LabelSelector `json:"nodeSelectors,omitempty"`
Expand Down Expand Up @@ -466,6 +466,7 @@ func (left *NodeState) Equal(right *NodeState) bool {
return reflect.DeepEqual(left, right)
}

// GetComplete returns a list of packages that are complete
func (ns *NodeState) GetComplete(packages Packages, interrupt map[string][]*Interrupt, config map[string][]string) []string {

ret := make([]string, 0)
Expand All @@ -488,6 +489,7 @@ func (ns *NodeState) GetComplete(packages Packages, interrupt map[string][]*Inte
return ret
}

// IsPackageComplete checks if a package is complete
func (ns *NodeState) IsPackageComplete(_package Package, interrupt map[string][]*Interrupt, config map[string][]string) bool {

packageStatus, found := (*ns)[_package.GetUniqueName()]
Expand All @@ -504,6 +506,7 @@ func (ns *NodeState) IsPackageComplete(_package Package, interrupt map[string][]
return false
}

// ProgressSkipped checks if a package is skipped and should be progressed to complete
func (ns *NodeState) ProgressSkipped(packages Packages, interrupt map[string][]*Interrupt, config map[string][]string) bool {
ret := false
for _, s := range *ns {
Expand Down Expand Up @@ -574,8 +577,8 @@ type State string
const (
METADATA_PREFIX string = "skyhook.nvidia.com"
StateComplete State = "complete"
StateInProgress State = "in_progress"
StateSkipped State = "skipped"
StateInProgress State = "in_progress" // this means its actually running, pod started
StateSkipped State = "skipped" // this means this package, stage are skipped mostly for some parts of the lifecycle
StateErroring State = "erroring"
StateUnknown State = "unknown"
)
Expand Down
22 changes: 1 addition & 21 deletions operator/api/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions operator/config/crd/bases/skyhook.nvidia.com_skyhooks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,8 +437,8 @@ spec:
concert without needing to delete to ad in discovery of the issue.
type: boolean
podNonInterruptLabels:
description: PodNonInteruptLabels are a set of labels we want to monitor
pods for whether they Interruptible
description: PodNonInterruptLabels are a set of labels we want to
monitor pods for whether they Interruptible
properties:
matchExpressions:
description: matchExpressions is a list of label selector requirements.
Expand Down
Loading
Loading