Resource reduction opportunities for gProfiler agent

[prashantbytesyntax]

When running gProfiler as a systemd service, we observe two key issues:

1. One Profiler trying to profile for the Same Target PID

Issue:
Multiple gProfiler-related processes are created to profile the same target process ID.
Example: Both PyPerf (each from staticx and pyinstaller) may attempt to profile the same Python process, leading to redundant profiler subprocesses and increased resource usage. Also noticed 2 subprocesses from gProfiler too ( from staticx installer)

Impact:

Wasted system resources (CPU, memory, file handles)
Potential contention or interference between profiler subprocesses
Harder to reason about profiling results and system load

2. Resource Cleanup: Ensure Profiler Subprocesses Are Properly Killed

Feature Request:
Each profiler subprocess (Java, Python, perf, etc.) is launched with a timeout and kill signal (SIGKILL/SIGTERM). However, we want an additional safety check to guarantee that all profiler subprocesses are actually terminated after each profiling duration.

Proposal:

After each profiling interval, use psutil to check for any lingering profiler subprocesses (e.g., py-spy, async-profiler, perf) that should have exited.
If any are found, forcibly terminate them using psutil.Process(pid).kill() and log the action.
Optionally, add a summary log or metric for "zombie" profiler subprocesses cleaned up in this way.

3. Multiple process of gProfiler spun up from staticx installer
4. Two profilers trying to profiler same target process

[mlim19]

Thanks. Will take a look.

[prashantbytesyntax]

#1 and #2 : this was due to memory leak where python parent process still have file descriptors open on its child process. Python GC does not have this visibility. We have the fix by making sure pipes are closed for terminated processes

#4 : gProfiler does not support mutual exclusion which is by design. So you will have a process profiled by both Java async profiler and perf - at profile collection time, logic decides what samples to choose based on scaling. However, there should be an option to support mutual exclusion.

#3 : Still waiting for explanation

We can close once we get explanation for #3.

[prashantbytesyntax]

thanks for helping with popen fix - #993

Additional resource reduction opportunities :

Perf Profiler Memory optimization

Investigation revealed that perf memory consumption (948MB observed) was primarily due to:
System-wide profiling: perf -a collects data from all processes
Text expansion: perf script converts binary data to text (10x size increase)
Python string processing: Large strings held in memory during parsing

python3 analyze_memory_detailed.py 🔍 Auto-detecting CSV format... ✅ Detailed memory plot saved to: gprofiler_detailed_memory_plot.png 📊 Current: 867.3 MB | Max: 1003.0 MB | Min: 0.0 MB 🔍 PEAK MEMORY ANALYSIS: Time: 2025-08-26 05:22:36 Total: 1003.0 MB Top memory consumers: 1. perf: 493.9 MB (505780 KB) 2. gprofiler: 120.9 MB (123832 KB) 3. PyPerf: 96.8 MB (99168 KB) 4. bash: 3.2 MB (3300 KB)
Tested Solution
Faster file rotation to avoid parsing on large files : https://github.com/pinterest/gprofiler/pull/20/files
Results : 15% improvement

Future Solution :
Improve string processing to stream based processing by avoid loading entire file into memory

Memory Explosion Analysis:

1. perf script output: 50-200MB of text
2. .decode("utf8"): Loads entire output as Python string (50-200MB)
3. .split("\n\n"): Creates list of all samples (100-400MB total)
4. String processing: Additional copies during regex matching (150-600MB peak) SAMPLE_REGEX.match(sample)

Perf outburst - 1k+ process

Issue
Memory exhaustion on high-process systems: 1.5k+ processes

Impact:
→ 2-3GB memory usage → OOM kills especially seen

Root cause :
gProfiler profiled ALL processes simultaneously without limits + continuous system profilers (perf/PyPerf) always running

Proposed solution :

1. Runtime Profiler limited by top N resource bound process
2. Disable System Profiler : unfortunately system profiler like perf crashes when pids are passed. So an immediate is to disable system profiler if over N process
3. Cgroup based filtering for Perf : Cgroup filtering would provide Crash-resistant system profiling with automatic process lifecycle management.

`
Robust - survives process churn
perf record --cgroup /sys/fs/cgroup/application.slice
perf record --cgroup /sys/fs/cgroup/user.slice

Benefits:

1. No crashes from missing PIDs
2. Automatic process inclusion/exclusion
3. Container-aware profiling
4. Resource-based filtering
   `

Suggestion to further breakdown cgroup Individual Docker container profiling instead of top level cgroup path.