[Bug]: Platform connector panics on shutdown

[lalitadithya]

Prerequisites

• I searched existing issues
• I can reproduce this issue

Bug Description

Platform connector panics on shutdown

Component

Health Monitor

Steps to Reproduce

1. Start platform connector
2. Tail logs
3. Kill pod and notice the error

Environment

• NVSentinel version: 0.3.0
• Kubernetes version: 1.33
• Deployment method: helm

Logs/Output

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x199d89d]

goroutine 30 [running]:
github.com/nvidia/nvsentinel/platform-connectors/pkg/connectors/kubernetes.(*K8sConnector).processHealthEvents(0xc000245bc0, {0x24b5a60, 0xc0002520f0}, 0x0)
	github.com/nvidia/nvsentinel/platform-connectors/pkg/connectors/kubernetes/process_node_events.go:327 +0x3d
github.com/nvidia/nvsentinel/platform-connectors/pkg/connectors/kubernetes.(*K8sConnector).FetchAndProcessHealthMetric(0xc000245bc0, {0x24b5a60, 0xc0002520f0})
	github.com/nvidia/nvsentinel/platform-connectors/pkg/connectors/kubernetes/k8s_connector.go:86 +0x7e
created by main.initializeK8sConnector in goroutine 1
	github.com/nvidia/nvsentinel/platform-connectors/main.go:109 +0x269

[lalitadithya]

@Gyan172004 , is this something you are interested in picking up?

[Gyan172004]

Sure @lalitadithya, I can work on it.
Looks like the shutdown sequence is not synced properly and a goroutine hits a nil event. I will look into it

[Gyan172004]

@lalitadithya

The nil check fixes the immediate panic (since Dequeue() returns nil when the queue shuts down).
But after diagnosing the logs and reproducing it, I found the real root cause is a race condition in our shutdown handling

Basically, it relies on timing, if the stop signal comes slightly later than the queue close, it crashes.

This fix works for this issue, but this race condition might cause other hidden issues later (like dropped events or partial processing).
PTAL if we might need to open a separate issue to refactor the shutdown logic properly so it’s deterministic and not just relying on lucky timing.

[lalitadithya]

Instead of doing an immediate patch, can we do the proper fix for the race?

[Gyan172004]

Sure, I will work on it