feat: change how limits are manged to a use a limitrange via helm

includes updates to documentation organization

Author: lockwobr

README.md

@@ -42,6 +42,7 @@ Skyhook works in any Kubernetes environment (self-managed, on-prem, cloud) and s
 - **Package Interrupt:** service (containerd, cron, any thing systemd), or reboot
 - **Additional Tolerations:**  are tolerations added to the packages
 - [**Runtime Required**](docs/runtime_required.md): requires node to come into the cluster with a taint, and will do work prior to removing custom taint.
+- **Resource Management:** Skyhook uses Kubernetes [LimitRange](https://kubernetes.io/docs/concepts/policy/limit-range/) to set default CPU and memory requests/limits for all containers in its namespace. You can override these defaults per-package in your Skyhook CR. Strict validation is enforced: if you set any resource override, you must set all four fields (cpuRequest, cpuLimit, memoryRequest, memoryLimit), and limits must be >= requests. See [docs/resource_management.md](docs/resource_management.md) for details and examples.
 
 ## Pre-built Packages
 
@@ -137,29 +138,13 @@ Part of how the operator works is the [skyhook-agent](agent/README.md). Packages
 └── config.json
 ```
 
-## Example Kyverno Policy
+## Examples
 
-This repository includes an example Kyverno policy that demonstrates how to restrict the images that can be used in Skyhook packages. While this is not a complete policy, it serves as a template that end users can modify to fit their security needs.
+See the [examples/](examples/) directory for sample manifests, usage patterns, and demo configurations to help you get started with Skyhook.
 
-The policy prevents the creation of Skyhook resources that contain packages with restricted image patterns. Specifically, it blocks:
-- Images containing 'shellscript:' anywhere in the image name
-- Images from Docker Hub (matching 'docker.io/*')
+## Kyverno Policy Examples
 
-If you are going to use kyverno make sure to turn on the creation of the skyhook-viewer-role in the values file for the operator. (rbac.createSkyhookViewerRole: true) and then bind kyverno to that role. Example policy:
-```
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: kyverno-skyhook-binding
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: skyhook-viewer-role
-subjects:
-- kind: ServiceAccount
-  name: kyverno-reports-controller
-  namespace: kyverno
-```
+See [docs/kyverno/README.md](docs/kyverno/README.md) for example Kyverno policies and guidance on restricting images or packages in Skyhook resources.
 
 ## [Skyhook-Operator](operator/README.md)
 The operator is a kbuernetes operator that monitors cluster events and coordinates the installation and lifecycle of Skyhook packages.

chart/README.md

@@ -35,4 +35,7 @@ Settings | Description | Default |
 ### NOTES
 - **estimatedPackageCount** and **estimatedNodeCount** are used to size the resource requirements. Default setting should be good for nodes > 1000 and packages 1-2 or nodes > 500 and packages >= 4. If your approaching this size deployment it would make sense to set these. You can also override them by explicitly with `controllerManager.manager.resources` the values file has an example.
 - **runtimeRequired**: If your systems nodes have this taint make sure to add the toleration to the controllerManager.tolerations
-- **CRD**: This project currently has one CRD and its not managed the ["recommended" way](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/). Its part of the templates. Meaning it will be updated with the `helm upgrade`. We decided it was better do it this way for this project. Doing it either way has consequences and this route has worked well for upgrades so far our deployments.
+- **CRD**: This project currently has one CRD and its not managed the ["recommended" way](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/). Its part of the templates. Meaning it will be updated with the `helm upgrade`. We decided it was better do it this way for this project. Doing it either way has consequences and this route has worked well for upgrades so far our deployments.
+
+### Resource Management
+Skyhook uses Kubernetes LimitRange to set default CPU/memory requests/limits for all containers in the namespace. You can override these per-package in your Skyhook CR. Strict validation is enforced. See [../docs/resource_management.md](../docs/resource_management.md) for details and examples.

chart/templates/limitrange.yaml

@@ -0,0 +1,16 @@
+{{- if .Values.limitRange }}
+apiVersion: v1
+kind: LimitRange
+metadata:
+  name: {{ include "skyhook.fullname" . }}-default-limits
+  namespace: {{ .Release.Namespace }}
+spec:
+  limits:
+    - type: Container
+      default:
+        cpu: {{ .Values.limitRange.default.cpu | default "500m" }}
+        memory: {{ .Values.limitRange.default.memory | default "512Mi" }}
+      defaultRequest:
+        cpu: {{ .Values.limitRange.defaultRequest.cpu | default "250m" }}
+        memory: {{ .Values.limitRange.defaultRequest.memory | default "256Mi" }}
+{{- end }}

chart/values.yaml

@@ -122,4 +122,17 @@ webhook:
 
   ## uninstall image for cleaning up webhook resources
   removalImage: bitnami/kubectl
-  removalTag: latest
+  removalTag: latest
+
+## limitRange: is the limit range for the operator controller.
+## This sets for all containers in the namespace.
+## So if your package does not override the limits, these are what will be used.
+## if you omit this, the we will not create a limit range.
+## best practice on limits and requests is to make make the limits 2x the requests max.
+limitRange:
+  default:
+    cpu: "500m"
+    memory: "512Mi"
+  defaultRequest:
+    cpu: "250m"
+    memory: "256Mi"

docs/README.md

@@ -0,0 +1,20 @@
+# Skyhook Documentation
+
+This directory contains user and operator documentation for Skyhook. Here you'll find guides, examples, and reference material to help you deploy, configure, and secure Skyhook in your Kubernetes cluster.
+
+## Available Documentation
+
+- [Resource Management](resource_management.md):
+  How Skyhook manages CPU/memory resources using LimitRange, per-package overrides, and validation rules.
+
+- [Kyverno Policy Examples](kyverno/README.md):
+  Example Kyverno policies for restricting images or packages in Skyhook resources.
+
+- [Providing Secrets to Packages](providing_secrets_to_packages.md):
+  How to securely provide secrets to Skyhook-managed packages.
+
+- [Releases](releases.md):
+  Release notes and upgrade information for Skyhook.
+
+- [Runtime Required](runtime_required.md):
+  How to use the runtime required taint and feature in Skyhook.

docs/kyverno/README.md

@@ -0,0 +1,12 @@
+# Kyverno Policy Examples for Skyhook
+
+This directory contains example [Kyverno](https://kyverno.io/) policies for use with Skyhook. These are **not installed by default** and are provided as templates for users to adapt to their own security needs.
+
+- `disable_packages.yaml`: Example policy to restrict or disable certain Skyhook packages/images.
+- `skyhook-viewer-binding.yaml`: Example RBAC binding for Kyverno to view Skyhook resources.
+
+**Note:**
+- This directory was previously at the repo root and has been moved to `docs/kyverno/` for clarity.
+- If you use these policies, ensure you enable the `skyhook-viewer-role` in your Helm values and bind Kyverno to that role.
+
+See the main [README](../README.md) for more information about Skyhook. 

kyverno/README.md renamed to docs/kyverno/kyverno/README.md

@@ -0,0 +1,111 @@
+# Resource Management in Skyhook
+
+Skyhook provides flexible and robust resource management for the pods it creates, allowing you to control CPU and memory usage at both the namespace and per-package level. This document explains how resource defaults and overrides work, and what validation rules are enforced.
+
+---
+
+## 1. Namespace Defaults with LimitRange
+
+By default, Skyhook uses a [Kubernetes LimitRange](https://kubernetes.io/docs/concepts/policy/limit-range/) to set default CPU and memory requests/limits for all containers in the namespace where Skyhook operates.
+
+**Example LimitRange:**
+```yaml
+apiVersion: v1
+kind: LimitRange
+metadata:
+  name: skyhook-default-limits
+  namespace: <your-namespace>
+spec:
+  limits:
+    - type: Container
+      default:
+        cpu: 500m
+        memory: 512Mi
+      defaultRequest:
+        cpu: 250m
+        memory: 256Mi
+```
+- If a pod/container does **not** specify its own resources, these defaults are applied.
+- You can configure these values via the Helm chart or Kustomize overlays.
+
+---
+
+## 2. Per-Package Resource Overrides
+
+You can override the default resource requests/limits for each package in your Skyhook Custom Resource (CR). This is done in the `resources` field for each package:
+
+**Example:**
+```yaml
+spec:
+  packages:
+    mypackage:
+      version: 1.0.0
+      image: ghcr.io/nvidia/skyhook-packages/shellscript
+      resources:
+        cpuRequest: "200m"
+        cpuLimit: "400m"
+        memoryRequest: "128Mi"
+        memoryLimit: "256Mi"
+```
+- If **any** of the four fields (`cpuRequest`, `cpuLimit`, `memoryRequest`, `memoryLimit`) are set, **all four must be set** and must be positive values.
+- If no override is set, the namespace's LimitRange applies.
+
+---
+
+## 3. Validation Rules
+
+Skyhook enforces the following validation rules (via webhook) for resource overrides:
+
+- If any of the four resource fields are set, **all four must be set**.
+- All values must be **positive**.
+- `cpuLimit` must be **greater than or equal to** `cpuRequest`.
+- `memoryLimit` must be **greater than or equal to** `memoryRequest`.
+
+**Examples:**
+
+| Valid? | cpuRequest | cpuLimit | memoryRequest | memoryLimit | Reason |
+|--------|-----------|----------|--------------|-------------|--------|
+| ✅     | 200m      | 400m     | 128Mi        | 256Mi       | All set, valid |
+| ❌     | 200m      |          | 128Mi        | 256Mi       | Not all fields set |
+| ❌     | 200m      | 100m     | 128Mi        | 256Mi       | cpuLimit < cpuRequest |
+| ❌     | 200m      | 400m     | 128Mi        | 64Mi        | memoryLimit < memoryRequest |
+| ❌     | 0         | 400m     | 128Mi        | 256Mi       | Zero value |
+
+If a resource override is invalid, the Skyhook CR will be **rejected** by the webhook.
+
+---
+
+## 4. Best Practices
+
+- Use LimitRange to set sensible defaults for your namespace.
+- Only set per-package overrides if you need different resource requirements for a specific package.
+- Review your resource settings to avoid overcommitting or underutilizing cluster resources.
+- If you change LimitRange defaults, new pods will use the new defaults unless overridden.
+
+---
+
+## 5. Troubleshooting
+
+- If your Skyhook CR is rejected, check that all four resource fields are set and valid if you are using overrides.
+- Use `kubectl describe limitrange -n <namespace>` to see the current defaults.
+- Use `kubectl describe skyhook <name>` to see the status and any error messages.
+
+---
+
+## 6. Disabling Resource Defaults (No Limits)
+
+If you do **not** want any default resource requests or limits applied to your Skyhook-managed pods/containers, you can simply **omit the LimitRange** from your namespace:
+
+- **Helm:** Set `limitRange: {}` or remove the `limitRange` section from your `values.yaml`.
+
+If there is **no LimitRange** and you do **not** set resource requests/limits in your package overrides, then:
+- Your pods/containers will run with **no resource requests or limits**.
+- This means they will be scheduled as "BestEffort" pods, which may be evicted first under resource pressure and may not get guaranteed CPU/memory.
+
+**Note:**
+- Disabling resource limits is not recommended for production clusters, as it can lead to resource contention and unpredictable scheduling.
+- Only do this if you have a specific reason and understand the implications.
+
+---
+
+For more information, see the [Kubernetes documentation on resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) and [LimitRange](https://kubernetes.io/docs/concepts/policy/limit-range/).