Numaflow Kafka Sourcer/Sinker

Overview

Numaflow Kafka Sourcer/Sinker is a Numaflow user-defined source/sink connector for Apache Kafka. It allows you to read/write data from/to a Kafka topic using Numaflow. Integration with Confluent Schema Registry is also supported.

Use Cases

Read data from Kafka

Use Case 1: Read data from Kafka with an Avro schema registered in the Confluent Schema Registry. See an example here.

Use Case 2: Read data from Kafka with no schema or JSON schema registered in the Confluent Schema Registry. See an example here.

Write data to Kafka

Use Case 3: Write data to Kafka with an Avro schema registered in the Confluent Schema Registry. See an example here.

Use Case 4: Write data to Kafka with a JSON schema registered in the Confluent Schema Registry. See an example here.

Use Case 5: Write data to Kafka with no schema. See an example here.

FAQ

How do I configure logging level?

Set environment variables in your container spec. Default level is INFO if not specified.

env:
  # Set root level (affects all packages)
  - name: LOGGING_LEVEL_ROOT
    value: "WARN"
  # Set level for io.numaproj.kafka.consumer package
  - name: LOGGING_LEVEL_IO_NUMAPROJ_KAFKA_CONSUMER
    value: "DEBUG"

Available levels: TRACE, DEBUG, INFO (default), WARN, ERROR, OFF

How do I enable structured logging (JSON)?

Structured logging is supported out of the box. To enable it, set the following environment variable in your container spec:

env:
  - name: LOGGING_STRUCTURED_FORMAT_CONSOLE
    value: "logstash"

Value	Format
logstash	Logstash JSON
ecs	Elastic Common Schema