Merge pr155 some important commits

[maverick123123]

---

Commit 1: Add seqlock for precise memory accounting (c2ae4aa)

Problem: The existing lock-free atomic reads used memory_order_relaxed for reading per-process memory counters. While individual atomic loads are safe, a reader could see a torn snapshot — e.g., reading total
updated by one allocation but context_size not yet updated — leading to inconsistent memory accounting and potentially incorrect OOM decisions.

Solution: Introduces a per-process sequence lock (seqlock) — a _Atomic uint64_t seqlock field added to shrreg_proc_slot_t in the header.

• Writer side (add_gpu_device_memory_usage, rm_gpu_device_memory_usage): Before updating counters, atomically increments seqlock to an odd value (signaling "write in progress"), performs all counter updates with
  memory_order_release, then increments seqlock to even (signaling "write complete"). This applies to both the fast path (cached my_slot pointer for own PID) and the slow path (linear scan for other PIDs).
• Reader side (get_gpu_memory_usage): For each process slot, reads seq1, reads the memory counter, inserts an acquire fence, reads seq2. If seq1 != seq2 (write happened during read), retries. If seq1 is odd
  (writer active), spins with exponential backoff: first 5 retries use CPU pause instructions (pause on x86, yield on ARM), then 1μs delays, then 10μs, then 100μs after 100 retries with debug logging.
• Header change: Added _Atomic uint64_t seqlock to shrreg_proc_slot_t, reduced unused[3] to unused[2] to maintain struct size/alignment. Seqlock is initialized to 0 (even = no write) in init_proc_slot_withlock.

---

Commit 2: Critical fix — Stale semaphore timeouts causing 300s hangs (fb4f906)

Problem: Processes would hang for 300+ seconds waiting for lock_shrreg(). Root cause: get_timespec() was called once before the while loop. After the first sem_timedwait() timeout, the timestamp was already in
the past, so every subsequent call returned ETIMEDOUT immediately — the process burned through all 30 retries in milliseconds with no actual waiting, then force-posted the semaphore. sem_post() on an
already-unlocked semaphore (value=1) made it value=2, allowing two processes to enter the critical section simultaneously, corrupting delta detection in set_task_pid().

Solution:

1. Moved get_timespec() inside the while loop in both lock_shrreg() and the new lock_postinit() — each iteration gets a fresh absolute timeout.
2. New lock_postinit() / unlock_postinit() functions using a dedicated sem_postinit semaphore for serializing host PID detection during postInit(). Uses longer timeouts: SEM_WAIT_TIME_POSTINIT=30s per wait,
   SEM_WAIT_RETRY_TIMES_POSTINIT=10 retries (total 300s max), since set_task_pid() with adaptive NVML polling can take several seconds.
3. Graceful timeout handling: lock_postinit() returns 1 on success, 0 on timeout. The caller in libvgpu.c:postInit() checks the return value and skips host PID detection on timeout instead of force-posting the
   semaphore. This prevents semaphore corruption entirely.
4. Header/init changes: Added sem_t sem_postinit to shared_region_t, initialized via sem_init() in try_create_shrreg().

---

Commit 3: Make exit cleanup foolproof — atomic operations only, no semaphore needed (46399f2)

Problem: The old exit_handler() tried to acquire region->sem (with a 5s timeout) to clean up its process slot. When 8 processes exited simultaneously, semaphore contention was high — some processes timed out,
leaving stale owner_pid values and locked semaphores. The next run would see a dead owner and get stuck. Exit cleanup that can fail defeats its purpose.

Solution — complete redesign of exit_handler():

1. No semaphore acquisition at all. Instead:
   - Atomically checks if this process holds owner_pid using atomic_load. If yes, uses CAS (atomic_compare_exchange_strong) to clear it to 0, then posts the semaphore to unlock it. CAS ensures only the actual
   owner clears it.
   - Atomically sets its process slot's PID to 0 (atomic_store_explicit with memory_order_release) and status to 0, marking the slot as dead. This is a simple, contention-free operation that cannot fail.
2. Lazy slot cleanup: Dead slots (PID=0) are not physically removed during exit. Instead, clear_proc_slot_nolock() was enhanced to detect and remove PID=0 slots (from exit cleanup) in addition to checking
   proc_alive() for truly dead processes. This runs when the next process acquires the lock during initialization. Added a limit of 10 proc_alive() checks per call to avoid holding the lock too long.
3. SIGKILL recovery in lock_shrreg(): SIGKILL is the only signal that bypasses the exit handler. On ETIMEDOUT, the code now uses atomic_load to read owner_pid, checks proc_alive(), and if the owner is dead, uses
   CAS to atomically clear owner_pid and post the semaphore. Only one process performs recovery thanks to CAS.
4. Hard deadlock protection: After 30 retries (5 minutes), lock_shrreg() logs an error with diagnostic info (owner alive/dead) and exits with an actionable message: "Delete /tmp/cudevshr.cache and restart all
   processes."

Guarantees: Normal exit, signal exit (SIGTERM/SIGINT/etc.), and SIGKILL all result in clean state — either immediately or on the next process's lock acquisition.

---

Commit 4: Optimize seqlock and utilization watcher to prevent random 256MB allocation slowdowns (c67cbbe)

Problem: Random 20x performance variance observed (12.734ms vs 0.586ms for 256MB allocations) when all 8 processes allocate simultaneously. Two root causes:

1. Seqlock retry storm: When 8 processes all write to their slots, readers see active writers (odd seqlock) and spin in a tight loop, causing CPU contention that slows down the writers further (a feedback loop).
2. Utilization watcher contention: The get_used_gpu_utilization() function in multiprocess_utilization_watcher.c held lock_shrreg() while making slow NVML queries (nvmlDeviceGetComputeRunningProcesses,
   nvmlDeviceGetProcessUtilization), blocking all shared memory operations for the entire duration.

Solution:

1. Seqlock exponential backoff (already described in Commit 1 — the backoff was added/refined in this commit): Removed the old "fallback to stale data after 100 retries" approach since memory checks require
   accurate data. Replaced with progressive delays: CPU pause instructions → 1μs → 10μs → 100μs. This reduces CPU contention while ensuring accurate reads.
2. Utilization watcher lock scope reduction: Restructured get_used_gpu_utilization() to perform NVML queries outside the lock. For each device:
   - Call nvmlDeviceGetComputeRunningProcesses() and nvmlDeviceGetProcessUtilization() without holding the lock
   - Then acquire lock_shrreg() briefly only to update shared memory with the results
   - Unlock immediately after updates

This changes the lock scope from "entire loop over all devices including NVML calls" to "brief update of shared memory per device", reducing lock hold time from milliseconds to microseconds.

---

Commit 5: Merge commit (2175e14)

Merge commit consolidating all changes into the testpr155 branch.

---

Architecture Summary

The PR transforms HAMi-core's multi-process GPU memory management from a design with several race conditions and failure modes into a robust system with:

• Seqlocks for consistent multi-field reads without blocking writers
• Fresh timeouts on every semaphore wait iteration (fixing the root cause of 300s hangs)
• Atomic-only exit cleanup that cannot fail or cause contention
• CAS-based SIGKILL recovery so only one process handles dead-owner detection
• Dedicated sem_postinit semaphore for host PID detection serialization
• Minimized lock hold times by moving NVML queries outside critical sections

[archlitchi]

/lgtm

[hami-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: archlitchi, maverick123123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [archlitchi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment