dockerfile: install libzstd-dev for kafka zstd compression (4.2 backport)

[dbottini2]

Backport of #12003 to the 4.2 branch.

Container images were built without libzstd-dev in the builder stage, so
librdkafka's find_package(ZSTD) failed and WITH_ZSTD was disabled. Kafka
producers in the shipped images could not use zstd compression.

Changes:

• dockerfiles/Dockerfile: add libzstd-dev to the builder and debug builder
  stages so librdkafka is compiled with zstd support.
• tests/runtime/out_kafka.c: add a runtime test asserting librdkafka accepts
  compression.codec=zstd (only true when WITH_ZSTD is compiled in).

Addresses #11366 (Packaging: Enable ZSTD compression by default).

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] local-build-all.sh builds the native distro packages under packaging/distros/. This PR only changes the container images (dockerfiles/Dockerfile), which that script does not build. The container image build was verified instead (see test evidence above).
• Set ok-package-test label to test for all targets (requires maintainer to do).

Test evidence (4.2 branch)

Example config (rdkafka.compression.codec=zstd):

[INPUT]
    Name    dummy
    Tag     test

[OUTPUT]
    Name                       kafka
    Match                      test
    Brokers                    127.0.0.1:9092
    Topics                     test
    rdkafka.compression.codec  zstd

Container build (dockerfiles/Dockerfile) — zstd is now detected so librdkafka
is built with WITH_ZSTD:

-- Found ZSTD: /usr/lib/aarch64-linux-gnu/libzstd.so
WITH_ZSTD:BOOL=ON

Runtime test (tests/runtime/out_kafka.c::zstd_compression_available):

Test zstd_compression_available...              [ OK ]
SUCCESS: All unit tests have passed.

(On master the same test was additionally run under Valgrind with no leaks and 0 errors.)

Backporting

• This is the backport to the 4.2 release branch.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 92bcec9b-1bb2-4fbf-aa30-479505982ed9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cbca3a9072

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Guard zstd-only Kafka runtime test

In runtime-test builds where FLB_OUT_KAFKA is enabled but librdkafka was configured without WITH_ZSTD (for example, non-Docker source builds without a system libzstd-dev, or builds that explicitly disable zstd), this assertion now fails even though the Kafka output can still build and run with other codecs. The Dockerfile change only fixes the container builder stages, while cmake/kafka.cmake does not force librdkafka to use Fluent Bit's bundled zstd, so this generic runtime test makes supported Kafka test configurations fail unless zstd happens to be available.

Useful? React with 👍 / 👎.