Add out_kafka output

[salekseev]

Could be based on https://github.com/edenhill/librdkafka, payload should be JSON or MessagePack.

[edsiper]

Added to the to-do list. thanks

[panda87]

Kafka output is very useful feature since in our case every log coming through kafka and then reroute to it's stateful service like elasticsearch / hadoop / other real time streaming.
Actually this is the only feature im waiting for it to use FluentBit since I want to use Docker with fluentd log driver, and then forward it to kafka.

@edsiper do you may have eta when you will be able to start working on it?

[edsiper]

@panda87 between Q2/Q3

[solsson]

Is there ongoing work with this? There is a https://github.com/samsung-cnct/fluent-bit-kafka-output-plugin but it needs more work, configurability in particular. If a native impl is on its way I could try to contribute there instead.

[edsiper]

@solsson not implemented yet, as said, between Q2/Q3.

[solsson]

Yes I read so and interpreted it as 2017-06-30T23:59:59 :) So I figured there's some source somewhere. But I will give the go impl a chance.

[edsiper]

FYI:

On GIT Master I've pushed a new Kafka REST output plugin.

The plugin is still under development but functional, any feedback is welcome. You can find a configuration example here

[panda87]

Thanks @edsiper
What are the benefits to work against Kafka REST instead of the solid TCP?

[solsson]

On GIT Master I've pushed a new Kafka REST output plugin.

Interesting. I'll give it a try once there's a new build of fluent/fluent-bit:0.12-dev.

[edsiper]

@panda87 : both plugins will be implemented: kafka-rest and kafka (native TCP), this is a snapshot of kafka-rest

@solsson fluent/fluent-bit:0.12-dev it's under build process (it should take ~20 min)

[panda87]

Thanks @edsiper