Allow passing files other than ".conf" via "extraFiles" in Fluentbit configuration

[mkielar]

Context

I'm trying to pass a Lua script, to use by FluentBit. to do that, I passed the lua file through extraFiles property.
The problematic part is how the Helm Chart generated the fluent-bit.conf:

data:
  fluent-bit.conf: |
    {{- .Values.containerLogs.fluentBit.config.service  | nindent 4 }}
    {{- range $key, $val := .Values.containerLogs.fluentBit.config.extraFiles }}
    @INCLUDE {{ $key }}
    {{- end }}

This implementation causes all files in extraFiles collection to be treated the same way, and added to fluent-bit.conf using @INCLUDE directive. Needless to day, including a Lua script this way it breaks entire configuration.

Feature request

Can we add scripts property in containerLogs.fluentBit.config to allow passing files that are not supposed to be embedded in fluent-bit.conf?

[mkielar]

Potential workaround

Using code in Lua Filter seems to be a working workaround:

[FILTER]
  Name    lua
  Match   application.*
  call    generate_log_keys
  code    function generate_log_keys(tag, timestamp, record) ...

It's terrible for maintenance, though. I'd rather have the script in a separate file as it helps with code completion, unit-testing, etc.

[jmtt89]

Hi, I think it would be much easier to use the official Fluent Bit Helm chart, even other features like GeoIP are not supported either.

[mkielar]

@jmtt89 Using official FluentBit would make it possible, but I don't agree it would be easier. I'm a newcomer to EKS and since AWS promotes its CloudWatch Observability Addon as one-off to deploy full integration and I think it should allow more than just a basic setup. Passing additional files is not a big change, its actually the same mechanism as extraFiles, just without embedding them in fluent-bit.conf.