Skip to content

Segmentation fault in rd_kafka_poll when running Fluentd 1.19.2 #1051

Open
Open
Segmentation fault in rd_kafka_poll when running Fluentd 1.19.2#1051
Labels
bugSomething isn't working
Milestone
v6.0.4
@shubher13

Description

@shubher13
shubher13
opened on May 14, 2026

Describe the bug

Fluentd is run with multi-process workers and several @type rdkafka2 outputs (from fluent-plugin-kafka). Under load, the Ruby VM aborts with:

[BUG] Segmentation fault at 0x0000000000000000

The faulting stack is entirely inside rdkafka-0.21.0 (native_kafka.rb → rd_kafka_poll) and librdkafka.so, via FFI (ffi gem)

To Reproduce

  1. Install/run Fluentd with gems as below (fluent-plugin-kafka, rdkafka, ffi).
  2. Use workers 4 in (see configuration snippet).
  3. Configure multiple @type rdkafka2 matches (e.g. separate labels for log/fm/pm pipelines) plus optional heartbeatkafka → copy → rdkafka2 store.
  4. Generate production-like log throughput.
  5. Observe kubectl logs (stderr): Ruby [BUG] crash with rd_kafka_poll.

Expected behavior

No process-level segfault-class/ Ruby VM [BUG] abort.

Your Environment

- Fluentd version: 1.19.2
- Package version: 6.0.1
   rdkafka (0.21.0)
   ffi (1.17.0)
   fluent-plugin-kafka (0.19.5)
   fluentd (1.19.2)
   ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +PRISM [x86_64-linux]

- Operating system:
NAME="Rocky Linux"
VERSION="9.7 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.7"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.7 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
VENDOR_NAME="RESF"
VENDOR_URL="https://resf.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.7"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.7"

- Kernel version: 5.14.0-427.94.1.el9_4.x86_64

Your Configuration

    <system>
      workers 4
      <log>
        format json
        time_format %Y-%m-%dT%H:%M:%S.%6NZ
      </log>
      ignore_same_log_interval 30s
    </system>
    <source>
      @type http
      @id healthcheck_ipv4
      port 9880
      bind 0.0.0.0
    </source>
    <source>
      @type http
      @id healthcheck_ipv6
      port 9880
      bind "::"
    </source>
    <source>
      @type http
      @log_level error
      @id input_http_ipv4
      port 9000
      bind 0.0.0.0
      <transport tls>
        ca_path <<isroot_cert>>
        cert_path <<fluentd_cert>>
        private_key_path <<fluentd_key_cert>>
      </transport>
    </source>

    <source>
      @type http
      @log_level error
      @id input_http_ipv6
      port 9000
      bind ::
      <transport tls>
        ca_path <<isroot_cert>>
        cert_path <<fluentd_cert>>
        private_key_path <<fluentd_key_cert>>
      </transport>
    </source>

    <source>
      @type exec
      tag reload
      command sh /opt/scripts/reload-config-and-crt.sh
      run_interval 60s
      keys none
      <parse>
        @type json
      </parse>
    </source>

    <match reload>
      @type stdout
      <format>
        @type json
      </format>
    </match>

    <source>
      @type exec
      tag validate
      command sh /etc/fluent/validation/fluentd_dry_run.sh
      run_interval 10s
      keys none
      <parse>
        @type json
      </parse>
    </source>    

    <match validate>
      @type stdout
      label @VALIDATE
      <format>
        @type json
      </format>
    </match>

@include /etc/fluent/filter.conf

@include /etc/fluent/logdata/logdata.conf

@include /etc/fluent/fmdata/fmdata.conf

@include /etc/fluent/fmdata/heartbeat-kafka.conf

@include /etc/fluent/pmdata/pmdata.conf



--------------------------------------------------------------------
eg:
logdata/log_kafka_label.conf

<label @KAFKALOG>
  <match {*security_logs*,*debug_logs*,*LI_logs*,*custom_logs*,*audit_logs*}>
    <buffer topic>
      @type file
      path /data/fluentdlogs/logskafka
      flush_thread_count 4
      chunk_limit_size 500KB
      overflow_action drop_oldest_chunk
      flush_mode interval
      flush_interval 5s
      total_limit_size 2GB
    </buffer>
    @type rdkafka2
    get_kafka_client_log false
    topic_key topic
    default_topic <REDACTED_DEFAULT_TOPIC>
    max_send_limit_bytes 1000000
    compression_codec none
    brokers "<REDACTED_BROKERS>"   # e.g. host:port; SSL lines commented in lab config
    <format>@type json</format>
    use_event_time false
    required_acks 1
  </match>
</label>

Your Error Log

/opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:53: [BUG] Segmentation fault at 0x0000000000000000
ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +PRISM [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0007 p:---- s:0027 e:000026 CFUNC  :rd_kafka_poll
c:0006 p:0009 s:0021 e:000020 BLOCK  /opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:53 [FINISH]
c:0005 p:---- s:0018 e:000017 CFUNC  :synchronize
c:0004 p:0007 s:0014 e:000013 BLOCK  /opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:52
c:0003 p:0017 s:0011 e:000010 METHOD <internal:kernel>:168
c:0002 p:0005 s:0006 e:000005 BLOCK  /opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:51 [FINISH]
c:0001 p:---- s:0003 e:000002 DUMMY  [FINISH]

-- Ruby level backtrace information ----------------------------------------
/opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:51:in 'block (2 levels) in start'
<internal:kernel>:168:in 'loop'
/opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:52:in 'block (3 levels) in start'
/opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:52:in 'synchronize'
/opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:53:in 'block (4 levels) in start'
/opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/native_kafka.rb:53:in 'rd_kafka_poll'

-- Threading information ---------------------------------------------------
Total ractor count: 1
Ruby thread count for this ractor: 53

-- Machine register context ------------------------------------------------
 RIP: 0x00007f8c41fce907 RBP: 0x00007f8b954d9dd0 RSP: 0x00007f8b954d9dc8
 RAX: 0x0000000000000000 RBX: 0x00007f8b954da1e0 RCX: 0x0000000400000000
 RDX: 0x0000000000000000 RDI: 0x0000000000000000 RSI: 0x0000000000000064
  R8: 0x00007f8c41fd6ae0  R9: 0x0000000000000000 R10: 0x0000000000000000
 R11: 0x00007f8c41fce900 R12: 0x0000000000000000 R13: 0x00007f8b954d9e80
 R14: 0x0000000002db6270 R15: 0x0000000000000001 EFL: 0x0000000000010246

-- C level backtrace information -------------------------------------------
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f820160) [0x7f8c5f820160]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f609540) [0x7f8c5f609540]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f763a12) [0x7f8c5f763a12]
/lib64/libc.so.6(__restore_rt+0x0) [0x7f8c5f1d2c30]
/opt/fluent/lib/ruby/gems/3.4.0/gems/rdkafka-0.21.0/lib/rdkafka/../../ext/librdkafka.so(rd_kafka_poll+0x7) [0x7f8c41fce907]
/lib64/libffi.so.8(0x7f8c43a0b8d6) [0x7f8c43a0b8d6]
/lib64/libffi.so.8(0x7f8c43a08556) [0x7f8c43a08556]
/opt/fluent/lib/ruby/gems/3.4.0/gems/ffi-1.17.0/lib/ffi_c.so(0x7f8c422f9469) [0x7f8c422f9469]
/opt/fluent/lib/libruby.so.3.4(rb_nogvl+0x2ae) [0x7f8c5f7b63ce]
/opt/fluent/lib/ruby/gems/3.4.0/gems/ffi-1.17.0/lib/ffi_c.so(rbffi_do_blocking_call+0x1c) [0x7f8c422f943c]
/opt/fluent/lib/libruby.so.3.4(rb_vrescue2+0x10b) [0x7f8c5f614b0b]
/opt/fluent/lib/libruby.so.3.4(rb_rescue2+0x8a) [0x7f8c5f614d9a]
/opt/fluent/lib/ruby/gems/3.4.0/gems/ffi-1.17.0/lib/ffi_c.so(rbffi_CallFunction+0x242) [0x7f8c422fa402]
/opt/fluent/lib/ruby/gems/3.4.0/gems/ffi-1.17.0/lib/ffi_c.so(0x7f8c422fdfd3) [0x7f8c422fdfd3]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f7ee56b) [0x7f8c5f7ee56b]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f8017e8) [0x7f8c5f8017e8]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f80801a) [0x7f8c5f80801a]
/opt/fluent/lib/libruby.so.3.4(rb_yield+0xc2) [0x7f8c5f80d1e2]
/opt/fluent/lib/libruby.so.3.4(rb_ensure+0xd9) [0x7f8c5f615099]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f7ee56b) [0x7f8c5f7ee56b]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f7f4c7d) [0x7f8c5f7f4c7d]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f8028c2) [0x7f8c5f8028c2]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f80801a) [0x7f8c5f80801a]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f808f97) [0x7f8c5f808f97]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f7b4825) [0x7f8c5f7b4825]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f7b4d52) [0x7f8c5f7b4d52]
/opt/fluent/lib/libruby.so.3.4(0x7f8c5f7b5ae4) [0x7f8c5f7b5ae4]
/lib64/libc.so.6(start_thread+0x31a) [0x7f8c5f21e2ea]
/lib64/libc.so.6(clone+0x44) [0x7f8c5f2a25f4]

-- Other runtime information -----------------------------------------------

* Loaded script: /usr/sbin/fluentd

* Loaded features:

    0 enumerator.so
    1 thread.rb
    2 fiber.so
    3 rational.so
    4 complex.so
    5 ruby2_keywords.rb
    6 /opt/fluent/lib/ruby/3.4.0/x86_64-linux/enc/encdb.so
    7 /opt/fluent/lib/ruby/3.4.0/x86_64-linux/enc/trans/transdb.so
    8 /opt/fluent/lib/ruby/3.4.0/x86_64-linux/rbconfig.rb
    9 /opt/fluent/lib/ruby/3.4.0/rubygems/compatibility.rb
   10 /opt/fluent/lib/ruby/3.4.0/rubygems/defaults.rb
   11 /opt/fluent/lib/ruby/3.4.0/rubygems/deprecate.rb
   12 /opt/fluent/lib/ruby/3.4.0/rubygems/errors.rb
   13 /opt/fluent/lib/ruby/3.4.0/rubygems/target_rbconfig.rb
   14 /opt/fluent/lib/ruby/3.4.0/rubygems/unknown_command_spell_checker.rb
   15 /opt/fluent/lib/ruby/3.4.0/rubygems/exceptions.rb
   16 /opt/fluent/lib/ruby/3.4.0/rubygems/basic_specification.rb
   17 /opt/fluent/lib/ruby/3.4.0/rubygems/stub_specification.rb
   18 /opt/fluent/lib/ruby/3.4.0/rubygems/platform.rb
   19 /opt/fluent/lib/ruby/3.4.0/rubygems/specification_record.rb
   20 /opt/fluent/lib/ruby/3.4.0/rubygems/util/list.rb
   21 /opt/fluent/lib/ruby/3.4.0/rubygems/version.rb
   22 /opt/fluent/lib/ruby/3.4.0/rubygems/requirement.rb
   23 /opt/fluent/lib/ruby/3.4.0/rubygems/specification.rb
   24 /opt/fluent/lib/ruby/3.4.0/rubygems/util.rb
   25 /opt/fluent/lib/ruby/3.4.0/rubygems/dependency.rb
   26 /opt/fluent/lib/ruby/3.4.0/rubygems/core_ext/kernel_gem.rb
   27 /opt/fluent/lib/ruby/3.4.0/x86_64-linux/monitor.so
   28 /opt/fluent/lib/ruby/3.4.0/monitor.rb
   29 /opt/fluent/lib/ruby/3.4.0/rubygems.rb
   30 /opt/fluent/lib/ruby/3.4.0/bundled_gems.rb
   31 /opt/fluent/lib/ruby/3.4.0/rubygems/path_support.rb
   32 /opt/fluent/lib/ruby/3.4.0/error_highlight/version.rb
   33 /opt/fluent/lib/ruby/3.4.0/error_highlight/base.rb
   34 /opt/fluent/lib/ruby/3.4.0/error_highlight/formatter.rb
   35 /opt/fluent/lib/ruby/3.4.0/error_highlight/core_ext.rb
   36 /opt/fluent/lib/ruby/3.4.0/error_highlight.rb
   37 /opt/fluent/lib/ruby/3.4.0/did_you_mean/version.rb
   38 /opt/fluent/lib/ruby/3.4.0/did_you_mean/core_ext/name_error.rb
   39 /opt/fluent/lib/ruby/3.4.0/did_you_mean/levenshtein.rb
   40 /opt/fluent/lib/ruby/3.4.0/did_you_mean/jaro_winkler.rb
   41 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checker.rb
   42 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checkers/name_error_checkers/class_name_checker.rb
   43 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checkers/name_error_checkers/variable_name_checker.rb
   44 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checkers/name_error_checkers.rb
   45 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checkers/method_name_checker.rb
   46 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checkers/key_error_checker.rb
   47 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checkers/null_checker.rb
   48 /opt/fluent/lib/ruby/3.4.0/did_you_mean/tree_spell_checker.rb
   49 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checkers/require_path_checker.rb
   50 /opt/fluent/lib/ruby/3.4.0/did_you_mean/spell_checkers/pattern_key_name_checker.rb
   51 /opt/fluent/lib/ruby/3.4.0/did_you_mean/formatter.rb
   52 /opt/fluent/lib/ruby/3.4.0/did_you_mean.rb
   53 /opt/fluent/lib/ruby/3.4.0/syntax_suggest/core_ext.rb
   54 /opt/fluent/lib/ruby/3.4.0/rubygems/bundler_version_finder.rb
   55 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/tsort/lib/tsort.rb
   56 /opt/fluent/lib/ruby/3.4.0/rubygems/vendored_tsort.rb
   57 /opt/fluent/lib/ruby/3.4.0/rubygems/request_set/gem_dependency_api.rb
   58 /opt/fluent/lib/ruby/3.4.0/rubygems/request_set/lockfile/parser.rb
   59 /opt/fluent/lib/ruby/3.4.0/rubygems/request_set/lockfile/tokenizer.rb
   60 /opt/fluent/lib/ruby/3.4.0/rubygems/request_set/lockfile.rb
   61 /opt/fluent/lib/ruby/3.4.0/rubygems/request_set.rb
   62 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/gem_metadata.rb
   63 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/delegates/specification_provider.rb
   64 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/errors.rb
   65 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/action.rb
   66 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/add_edge_no_circular.rb
   67 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/add_vertex.rb
   68 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/delete_edge.rb
   69 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/detach_vertex_named.rb
   70 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/set_payload.rb
   71 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/tag.rb
   72 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/log.rb
   73 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph/vertex.rb
   74 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/dependency_graph.rb
   75 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/state.rb
   76 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/modules/specification_provider.rb
   77 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/delegates/resolution_state.rb
   78 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/resolution.rb
   79 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/resolver.rb
   80 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo/modules/ui.rb
   81 /opt/fluent/lib/ruby/3.4.0/rubygems/vendor/molinillo/lib/molinillo.rb
   82 /opt/fluent/lib/ruby/3.4.0/rubygems/vendored_molinillo.rb
   83 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/activation_request.rb
   84 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/conflict.rb
   85 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/dependency_request.rb
   86 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/requirement_list.rb
   87 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/stats.rb
   88 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/set.rb
   89 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/api_set.rb
   90 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/composed_set.rb
   91 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/best_set.rb
   92 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/current_set.rb
   93 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/git_set.rb
   94 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/index_set.rb
   95 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/installer_set.rb
   96 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/lock_set.rb
   97 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/vendor_set.rb
   98 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/source_set.rb
   99 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/specification.rb
  100 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/spec_specification.rb
  101 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/api_specification.rb
  102 /opt/fluent/lib/ruby/3.4.0/rubygems/resolver/git_specification.rb
...
...
...

Additional context

  • rdkafka-ruby CHANGELOG documents segfault-class fixes from 0.22.0 onwards but fluentd-package 6.0.1 still points to rdkafka-ruby 0.21.0
  • we did not observe this issue with fluentd-package 6.0.0 (Ruby 3.4.5, rdkafka 0.21.0). The issue started appearing after upgrading to fluentd-package 6.0.1 (Ruby 3.4.7, still using rdkafka 0.21.0).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Fluentd Kanban
    Status
    To-Do

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions