docs: update Kafka MSK IAM authentication configuration #2294
Conversation
WalkthroughInput and output Kafka docs were reorganized and expanded for AWS MSK IAM authentication: prerequisites and configuration parameters were clarified, examples updated (including PrivateLink/custom DNS), region auto-detection noted, and a focused IAM policy with explicit ARNs and placeholders was added. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
pipeline/outputs/kafka.md (4)
5-5: Fix plugin reference: output doc incorrectly says “Kafka input plugin”.
Line 5 should refer to the Kafka output plugin (producer), not the input plugin.-In Fluent Bit 4.0.4 and later, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access. +In Fluent Bit 4.0.4 and later, the Kafka output plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
27-27: Typo:librdfkafka→librdkafka.
Line 27 has a misspelling that may confuse readers searching for the right library knobs.-Setting `rdkafka.log.connection.close` to `false` and `rdkafka.request.required.acks` to `1` are examples of recommended settings of `librdfkafka` properties. +Setting `rdkafka.log.connection.close` to `false` and `rdkafka.request.required.acks` to `1` are examples of recommended settings of `librdkafka` properties.
48-58: Example config useshost(and a malformed IP) instead ofbrokers.
Line 56 has192.1681.3:9092(missing dot) and the key should match the documented parameterbrokers.outputs: - name: kafka match: '*' - host: 192.1681.3:9092 + brokers: 192.168.1.3:9092 topics: test
191-191: Minor formatting: add space in “rawformat”.-#### Kafka configuration file with `raw`format +#### Kafka configuration file with `raw` format
kalavt
force-pushed
the
feature/aws-msk-iam-auth
branch
2 times, most recently
from
4db76b7 to
9101aab
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pipeline/outputs/kafka.md (1)
5-6: Doc bug: “Kafka input plugin” should be “Kafka Producer output plugin” here.
This is in the producer/output page and is likely a copy/paste slip.
🧹 Nitpick comments (1)
pipeline/inputs/kafka.md (1)
149-165: Config parameters table/examples: consider explicitly stating whetheraws_msk_iam_cluster_arnis no longer needed (and why).
Right now the table + examples implyrdkafka.sasl.mechanism=aws_msk_iamis sufficient, but users familiar with older docs will look for the removed ARN parameter and may be unsure what changed.Also applies to: 173-183
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
pipeline/inputs/kafka.md(1 hunks)pipeline/outputs/kafka.md(1 hunks)
🔇 Additional comments (2)
pipeline/inputs/kafka.md (1)
195-225: IAM permissions + ARN formats look solid and much safer (scoped resources + CLUSTER_UUID note).
The consumer actions list (DescribeGroup/AlterGroup) and the CLUSTER_UUID guidance are the right level of specificity.pipeline/outputs/kafka.md (1)
347-374: Producer IAM policy example is clear and appropriately scoped.
Nice improvement over overly-broadResource: "*"style examples.
Signed-off-by: Arbin <[email protected]>
Change 'Kafka input plugin' to 'Kafka output plugin' in the introduction Signed-off-by: Arbin <[email protected]>
kalavt
force-pushed
the
feature/aws-msk-iam-auth
branch
from
9101aab to
a767991
Compare
There was a problem hiding this comment.
Actionable comments posted: 4
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
pipeline/inputs/kafka.md(1 hunks)pipeline/outputs/kafka.md(2 hunks)
🔇 Additional comments (2)
pipeline/inputs/kafka.md (1)
199-227: Approved: IAM permissions policy is now correct.The policy correctly includes the cluster UUID in ARN paths and specifies all required actions for a consumer (Connect, DescribeTopic, ReadData, DescribeGroup, AlterGroup). The guidance to substitute REGION, ACCOUNT, CLUSTER_NAME, and CLUSTER_UUID is clear.
pipeline/outputs/kafka.md (1)
351-376: Approved: Producer IAM permissions policy is correct.The policy correctly specifies producer-only actions (Connect, DescribeTopic, WriteData) with cluster and topic ARNs that include the CLUSTER_UUID segment. The guidance to substitute placeholders is clear.
|
@alexakreizinger review request for you! |
|
I want to give this my full attention, but I have to finish early for the day for an appointment so I'm setting a reminder to review it on Monday 😄 |
alexakreizinger
left a comment
alexakreizinger
left a comment
There was a problem hiding this comment.
@kalavt thanks for opening a PR, and apologies for the delayed review—I'm making some suggestions in line with our style guide, and feel free to merge if everything looks good to you :)
Signed-off-by: Alexa Kreizinger <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
pipeline/inputs/kafka.md (1)
145-147: Reword repetitive sentence structure in Prerequisites section.The three consecutive sentences beginning with "You must" can be combined or restructured for better readability. Consider:
- To use AWS MSK IAM authentication, you must meet these requirements: - - - You must have access to an AWS MSK cluster with IAM authentication enabled. - - You must have valid AWS credentials (IAM role, access keys, or instance profile). - - You must have network connectivity to your MSK brokers. + To use AWS MSK IAM authentication, you must: + + - Have access to an AWS MSK cluster with IAM authentication enabled. + - Provide valid AWS credentials (IAM role, access keys, or instance profile). + - Have network connectivity to your MSK brokers.pipeline/outputs/kafka.md (1)
244-248: Reword repetitive sentence structure in Prerequisites section.Same issue as the input plugin documentation: three consecutive sentences beginning with "You must" reduce readability. Apply the same refactoring:
- To use AWS MSK IAM authentication, you must meet these requirements: - - - You must have access to an AWS MSK cluster with IAM authentication enabled. - - You must have valid AWS credentials (IAM role, access keys, or instance profile). - - You must have network connectivity to your MSK brokers. + To use AWS MSK IAM authentication, you must: + + - Have access to an AWS MSK cluster with IAM authentication enabled. + - Provide valid AWS credentials (IAM role, access keys, or instance profile). + - Have network connectivity to your MSK brokers.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
pipeline/inputs/kafka.md(1 hunks)pipeline/outputs/kafka.md(2 hunks)
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: kalavt
Repo: fluent/fluent-bit-docs PR: 2294
File: pipeline/inputs/kafka.md:147-168
Timestamp: 2025-12-12T14:30:10.698Z
Learning: In Fluent Bit v4.0.4+, when using AWS MSK IAM authentication (rdkafka.sasl.mechanism: aws_msk_iam), the rdkafka.security.protocol is automatically set to SASL_SSL and the AWS region is auto-detected from the broker hostname for standard MSK endpoints. The aws_msk_iam_cluster_arn parameter was removed - users only need to set rdkafka.sasl.mechanism: aws_msk_iam (and optionally aws_region for custom DNS/PrivateLink).
📚 Learning: 2025-12-12T14:30:10.698Z
Learnt from: kalavt
Repo: fluent/fluent-bit-docs PR: 2294
File: pipeline/inputs/kafka.md:147-168
Timestamp: 2025-12-12T14:30:10.698Z
Learning: In Fluent Bit v4.0.4+, when using AWS MSK IAM authentication (rdkafka.sasl.mechanism: aws_msk_iam), the rdkafka.security.protocol is automatically set to SASL_SSL and the AWS region is auto-detected from the broker hostname for standard MSK endpoints. The aws_msk_iam_cluster_arn parameter was removed - users only need to set rdkafka.sasl.mechanism: aws_msk_iam (and optionally aws_region for custom DNS/PrivateLink).
Applied to files:
pipeline/inputs/kafka.mdpipeline/outputs/kafka.md
🪛 LanguageTool
pipeline/inputs/kafka.md
[style] ~147-~147: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...e, access keys, or instance profile). - You must have network connectivity to your ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
pipeline/outputs/kafka.md
[style] ~248-~248: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...e, access keys, or instance profile). - You must have network connectivity to your ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🔇 Additional comments (8)
pipeline/inputs/kafka.md (4)
149-154: Configuration parameters accurately document simplified MSK IAM setup.The table correctly reflects the streamlined approach:
rdkafka.sasl.mechanismand optionalaws_regionare the only MSK IAM-specific parameters needed. The auto-detection behavior for standard endpoints is clearly documented.
156-175: Basic configuration example is correct and well-documented.The minimal example correctly shows only
rdkafka.sasl.mechanism: aws_msk_iam, with clear notes about region auto-detection and automatic security protocol configuration. The documentation accurately reflects the simplified setup.
177-189: Custom DNS/PrivateLink example correctly documents aws_region parameter usage.The example appropriately shows when to set
aws_regionfor non-standard broker hostnames. Configuration remains minimal and the guidance is clear.
191-240: AWS credentials chain and IAM policy are comprehensive and accurate.The credentials chain follows AWS best practices, and the IAM policy correctly specifies actions (Connect, DescribeTopic, ReadData, DescribeGroup, AlterGroup) with properly formatted ARNs including the CLUSTER_UUID. The clarification about locating the cluster UUID is helpful.
pipeline/outputs/kafka.md (4)
250-255: Configuration parameters table is consistent with input plugin and accurate.The same streamlined parameters are documented for the output plugin, maintaining consistency across both plugins.
257-301: Basic configuration examples for output plugin are correct and well-documented.The YAML and .conf examples correctly show minimal MSK IAM setup for producers, with accurate documentation of auto-detection and auto-configuration behaviors. Consistent with the input plugin documentation.
303-341: Custom DNS/PrivateLink examples for output plugin are accurate and complete.Both YAML and .conf formats correctly demonstrate aws_region usage for non-standard hostnames. Documentation aligns with the input plugin guidance.
353-389: Producer IAM policy is correctly scoped with appropriate actions and ARN format.The policy appropriately specifies producer-specific permissions (Connect, DescribeTopic, WriteData) with correctly formatted ARNs. CLUSTER_UUID requirement and guidance are clearly documented, matching the input plugin documentation standards.
|
Hi @alexakreizinger appreciate your efforts on review and update the docs I would say let's hold it bit to wait for version v4.2.1 release to avoid confusion users |
|
@kalavt looks like this is still waiting on the code pr to merge: fluent/fluent-bit#11270, so marking as such here. |
eschabell
added
waiting-on-code-merge
and removed
waiting-on-review
3 participants
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.