fix(chart): resolve kubernetes security scan violations for compliance

[t0mmylam]

Overview

This PR addresses failing checks from the helm chart security scan in preparation for the October 1st compliance requirement when these checks will become blocking instead of warnings.

Security Fixes Implemented

Container Security Context Hardening:

• Added allowPrivilegeEscalation: false to all containers
• Enabled readOnlyRootFilesystem: true for enhanced security
• Dropped ALL capabilities from containers (CKV_K8S_19, CKV_K8S_22, CKV_K8S_27, CKV_K8S_28, CKV_K8S_34, CKV_K8S_37, CKV_K8S_43)

Resource Management:

• Added CPU and memory limits/requests to prevent resource exhaustion (BC_K8S_9, BC_K8S_10, BC_K8S_11, BC_K8S_12)

Health Monitoring:

• Added liveness probes for container health checks (BC_K8S_7, BC_K8S_8)

Image Security:

• Updated image references with specific digests instead of tags (BC_K8S_13, BC_K8S_14, BC_K8S_39)

Namespace Security:

• Added explicit namespace declarations to all resources (CKV_K8S_21)
• Added validation to prevent deployment to 'default' namespace (BC_K8S_29, BC_K8S_35)

Admission Webhook Security:

• Configured cert-manager integration to address webhook configuration concerns
• Set allowWriteWebhookConfig: false for least-privilege access

Excluded Check

CKV_K8S_155: "Minimize ClusterRoles that grant control over validating or mutating admission webhook configurations"

• Added a skip for this check after discussion with the team due to limitations

Testing

• Updated all helm test configurations to include new security contexts
• All CI tests pass with the new security configurations

References

• Security scan report: https://nspect.nvidia.com/reports/launch/chart?nspectId=NSPECT-AOXJ-B73K&resourceId=672
• Prisma Cloud policy documentation: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/

[ayuskauskas]

Did you run this chart in a kind cluster and verify that operator-agent tests and some of the e2e tests still work?

[t0mmylam]

Did you run this chart in a kind cluster and verify that operator-agent tests and some of the e2e tests still work?

Yes, verified locally with e2e-tests

[ayuskauskas]

The release docs also need to be updated to set the digest. Until now we were using the chart == operator version. But with using digest we have to fetch that now and put it into the chart. It would be good to add a section about updating the chart with a command that transforms an image + tag into the multi-arch digest

[ayuskauskas]

Did you run this chart in a kind cluster and verify that operator-agent tests and some of the e2e tests still work?

Yes, verified locally with e2e-tests

The e2e tests don't actually run the operator in k8s though. It runs from a local process onto the api. You should do a test where you install the helm chart then go through and run the simple operator agent directly

kubectl create namespace skyhook
helm install -n skyhook skyhook ./chart
cd operator
../k8s-tests/operator-agent/setup.sh kind-worker setup
./bin/chainsaw test --test-dir ../k8s-tests/operator-agent/simple

Make sure the kind-worker node has the label (which it will if you have run other e2e tests already on it)

skyhook.nvidia.com/test-node: skyhooke2e