Analyze Logstash RSS memory with `glibc` and `jemalloc`

[mashhurs]

Logstash RSS grows over time on Linux/glibc while JVM heap stays bounded — native allocator retention?

TL;DR:

• if you are observing high RSS memory and Logstash is running on Linux/glib, use -XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=60000 JVM options which periodically trims the native memory back to OS. UnlockExperimentalVMOptions option isn't required if JDK is 21.0.3+.
• if you want to know where the memory arena allocations are happening, use jemalloc, set dirty_decay_ms:-1 (telling do not auto trim native memory to see where retains pages are left) profile to figure out. Memory allocation depends on pipeline definition (which plugins are used)

Summary

On Logstash 8.19.13 running on Linux (glibc), process RSS (RES) grows steadily over time and is never released back to the OS, eventually reaching many GB, even though the JVM heap is bounded (-Xms1g -Xmx1g, pipeline buffers on heap). Forcing a native trim (System.trim_native_heap) reclaims a large amount of RES, which points at native allocator retention rather than a JVM-heap leak. We're trying to confirm the mechanism and the right mitigation, and to rule out a true native leak.

Environment

• Logstash 8.19.13 (bundled JDK 21), pipeline.buffer.type: heap
• Linux x86_64, glibc (default allocator), no MALLOC_ARENA_MAX / LD_PRELOAD set, used default what glibc 2.39 version has.

Symptoms

• top/ps RES climbs monotonically and does not come back down at idle.
• JVM heap usage itself remains within -Xmx; no OOME from the Java heap.

What we have found so far

• Reproduced the retention on a Linux/glibc container: RES ratchets up and holds even after large GCs (heap freed, RES not returned).
• The Logstash Node API / Puma threads are short-lived (created per scrape, reaped); thread count is bounded — not a thread leak, though the create/destroy churn does allocate native thread stacks via glibc.
• Allocation profiling (async-profiler nativemem) at idle/light load is dominated by JIT compilation (transient, code-cache bounded); workload paths (AWS SDK, sockets) are tiny.
• System.trim_native_heap or JVM UnlockExperimentalVMOptions drops RES substantially (e.g., ~11.7 GB → ~4.7 GB), with VIRT/SHR roughly unchanged.
• jemalloc profiling (jeprof diff of dumps over time) with dirty_decay_ms:-1 (-1 means no native trim claim to make sure jeprof files are generated overtime): warmup window shows class loading, zlib inflate, etc. (jar reads) and compiler-arena churn; steady-state diff is net-negative (live native heap stable/shrinking) while process RES keeps rising

Methodology (how we're analyzing)

1. RSS-over-time sampling of the Logstash PID (/proc/<pid>/status VmRSS) to characterize the growth/sawtooth and compare configurations.
2. Allocator identification: /proc/<pid>/maps, LD_PRELOAD/MALLOC_* env to confirm glibc vs jemalloc.
3. jemalloc heap profiling (MALLOC_CONF=prof:true,lg_prof_sample,lg_prof_interval) with periodic dumps, then jeprof --base diffs to isolate retained (live) growth and its native call sites (e.g., deflate/inflate/ssl/malloc).
4. JVM-side cross-checks: _node/stats (heap committed vs used), GC counts, thread counts; optionally NMT.

Mitigations tested

• -XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=60000 (JDK 21 periodic native trim): RES fluctuates and trends down — effective.
• jemalloc via LD_PRELOAD (returns memory to OS more aggressively than glibc): UNDER EVALUATION.
• MALLOC_ARENA_MAX / periodic malloc_trim (System.trim_native_heap): known to reclaim RES.

Reproduction (containerized)

1. docker.elastic.co/logstash/logstash:8.19.13 on Linux/glibc, -Xms1g -Xmx1g.
2. Pipeline: a generator/network input + an output that exercises native paths (e.g., s3 with encoding => "gzip"); API over TLS; Metricbeat scraping :9600 every 10s.
   • (S3 creds/bucket redacted: access_key_id/secret_access_key/bucket provided via env, e.g. s3://<BUCKET>.)
3. Sample VmRSS over time; observe RES ratchet on glibc.
4. Compare against the same image with jemalloc preloaded and/or TrimNativeHeapInterval enabled.

SEE THE BELOW ANALYSIS COMMENT - #19248 (comment)

[mashhurs]

I have setup a pipeline with simple generator input where input data will be uploaded to S3:

Click me to see the pipeline definition

 input {
     generator {
       count   => 0
       threads => 2
       lines   => [ ... ]
     }
 }
 output {
     s3 {
         access_key_id => "${AWS_ACCESS_KEY_ID}"
         secret_access_key => "${AWS_SECRET_ACCESS_KEY}"
         region => "${AWS_REGION:us-east-1}"
         bucket => "logstash-mashhur-test-2"
         prefix => "logstash/%{+YYYY/MM/dd}"
         time_file => 3
         encoding => "gzip"
         codec => "json_lines"
     }
 }

• (ls-glibc) Logstash docker 8.19.13 LS ubi image, glibc 2.39 setup showed RSS continuously increased.
• (ls-glibc-trim) Logstash docker 8.19.13 LS ubi image, glibc 2.39 with -XX:+UnlockExperimentalVMOptions -XX:TrimNativeHeapInterval=60000 options setup showed RSS stays steady.
• (ls-jemalloc) LS docker 8.19.13, Ubuntu 24.04.4 LTS, jemalloc 5.3.0 showed that native memory will be auto trimmed because default decay time dirty_decay_ms: 10s (purges retained pages every 10s)

---

FURTHER ANALYSIS

I have setup both 8.9.0 and 8.19.13 LS versions, with jemalloc profiling enabled, set dirty_decay_ms:-1 (by default 10s where jemalloc trims native memory automatically at this cadence) to see if LS new versions introduced the behavior.

I used the pipeline definition above and monitored ~4h. Note that I had tested multiple time, sharing the ~4h run metrics only here. Bothe LS versions show RSS continuously increased overtime.

I have used async-profiler tool which has nativemem option to capture which allocated arenas are belong to what LS code components. What I found interesting is, most (~80%) arenas, which are retained freed pages, belong to AWS SDK, LS S3 output.

Click me to see nativemem code paths

    docker exec -e MALLOC_CONF= ls-jemalloc sh -c '
   grep -iE "aws|s3" /tmp/nm3.collapsed | sort -t" " -k2 -nr | head -3 | awk "{n=\$NF; \$NF=\"\"; print \"=== \" n \" bytes ===\"; gsub(/;/,\"\n  \"); print}" | grep -iE "===|aws|s3|ssl|crypto|socket|write|read|byte|http|deflate|Digest|Buffer" | head -40
   '
   === 918848 bytes ===
   java/lang/Thread.run_[0]
     java/lang/Thread.runWith_[0]
     java/util/concurrent/ThreadPoolExecutor$Worker.run_[0]
     java/util/concurrent/ThreadPoolExecutor.runWorker_[0]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/uploader.RUBY$block$upload_async$1_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/uploader.RUBY$method$upload$0_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3.RUBY$method$clean_temporary_file$0_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/temporary_file.RUBY$method$delete!$0_[1]
   === 918848 bytes ===
   java/lang/Thread.run_[0]
     java/lang/Thread.runWith_[0]
     java/util/concurrent/ThreadPoolExecutor$Worker.run_[0]
     java/util/concurrent/ThreadPoolExecutor.runWorker_[0]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/uploader.RUBY$block$upload_async$1_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/uploader.RUBY$method$upload$0_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3.RUBY$method$clean_temporary_file$0_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/temporary_file.RUBY$method$delete!$0_[1]
   === 918848 bytes ===
   java/lang/Thread.run_[0]
     java/lang/Thread.runWith_[0]
     java/util/concurrent/ThreadPoolExecutor$Worker.run_[0]
     java/util/concurrent/ThreadPoolExecutor.runWorker_[0]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/uploader.RUBY$block$upload_async$1_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/uploader.RUBY$method$upload$0_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3.RUBY$method$clean_temporary_file$0_[1]
     usr/share/logstash/vendor/bundle/jruby/$3_dot_1_dot_0/gems/logstash_minus_integration_minus_aws_minus_7_dot_2_dot_1_minus_java/lib/logstash/outputs/s3/temporary_file.RUBY$method$delete!$0_[1]

Key findings with this experiment

• RSS increase isn't only 8.19.13 but was historically on 8.9.0 as well. Most probably all Logstash versions behave same.
• Claiming native memory through JVM doesn't hurt and better result comparing to jemalloc auto native memory trim. Use -XX:TrimNativeHeapInterval=60000 (60s), no need UnlockExperimentalVMOptions as we observed this param is not experimental anymore - https://bugs.openjdk.org/browse/JDK-8325496.
• The retained pages are belong to S3 output plugin, most probably AWS SDK that nativemem code path tells upload method which has @transfer_manager = Aws::S3::TransferManager.new(client: bucket.client) which seems target - https://github.com/logstash-plugins/logstash-integration-aws/blob/24f35166801a2ef85e4ed046ea501aee235bc33e/lib/logstash/outputs/s3/uploader.rb#L36. I will try to figure out to capture deeper level of code path.